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Abstract 

In  order  to  handle  its  obligations,  the  Brazihan  Ministry  of  Defense  (MoD)  wiU  need  an 
information  system  capable  of  managing  logistics  information  from  aU  mihtary  services.  A 
project  to  develop  an  integrated  information  system  to  fit  the  requirements  of  different,  but 
connected,  organizations  has  inherent  challenges.  Differences  in  the  organizational  stmctures, 
cultures  and  political  aspects,  are  key  issues  to  be  observed  before  the  development  to  assure 
the  project’s  success.  The  same  is  applicable  when  trying  to  adapt  an  already  existing 
information  system  to  fiU  the  needs  of  another  organization.  In  the  new  organization,  it  is 
mandatory  to  assess  the  feasibihty  of  the  software’s  alternatives  available.  Alternatives  can  be 
to  adapt  an  existing  information  system  or  to  develop  a  completely  new  system.  This  research 
sought  to  develop  a  method  for  assessing  the  organizational,  cultural,  and  pohtical  considerations 
affecting  the  insertion  of  the  Integrated  Logistics  Information  System  (SILOMS),  developed  by 
the  Brazihan  Air  Force,  into  the  MoD.  The  research  develops  a  method  for  assisting  decision 
makers  in  assessing  the  risks  involved  in  the  implementation  of  an  informahon  system  in  the 
MoD. 
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DEVELOPMENT  OF  A  METHOD  FOR  ASSESSING  THE  ORGANIZATIONAL, 
CULTURAL,  AND  POLITICAL  CONSIDERATIONS  AFFECTING  THE 
INSERTION  OF  SILOMS  INTO  THE  MoD 


I.  Introduction 


General  Issue 

In  the  past  decade  Brazil  has  been  reengineering  processes  relating  to  government 
bureaucratic  organizations  and  the  economy  in  order  to  prepare  the  country  for  globalization 
and  shrinking  budgets.  During  the  last  two  presidential  mandates,  from  1994  to  2002,  Brazil’s 
government  has  led  structural  changes  to  assure  the  country  a  competitive  place  in  the  new 
world  scenario.  Changes  have  been  made  within  all  government  departments’  stmcture  to 
decrease  expenditures  and  improve  efficiency  while  performing  an  increasing  number  of 
functions  to  deal  with  the  changing  environment. 

The  defense  system  administration  has  implemented  a  new  organizational  stmcture 
combining  the  former  three  separate  ministries  for  each  branch  of  mihtary  services  and  other 
defense  organizations  into  a  single  Ministry  of  Defense  (MoD)  -  see  Figure  1 .  The  former 
organizational  stmcture  was  considered  inefficient,  expensive,  and  was  not  integrated  across  the 
services.  Until  1995  Brazil’s  defense  system  was  based  upon  each  service  having  its  own 
Ministry  in  addition  to  an  overall  Ministry  of  the  Major  Staff  of  the  Armed  Forces.  Each  armed 
force  and  the  major  staff  had  its  own  bureaucratic  stmcture  and  was  responsible  to  perform  all 
affairs  related  to  a  Ministry.  This  proved  to  be  highly  expensive  and  inefficient.  A  lack  of  joint 
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effort  was  clearly  visible,  as  each  service  could  be  developing  a  program  or  weapon  system 
acquisition  without  considering  the  effort  already  being  done  by  other  services.  Duplication  of 
efforts  occurred,  and  many  opportunities  to  improve  joint  programs  were  lost  during  the  past 
decades  due  to  the  old  defense  organizational  stmcture. 


Figure  1.  Brazil’s  Ministry  of  Defense  Stmcture. 


As  the  new  concept  of  defense  organization  was  being  implemented  problems  came  up 
under  the  new  MoD  stmcture.  As  a  result  of  the  new  scenario,  specific  defense  system 
regulations  had  to  be  developed  while  old  ones  were  revised  as  the  armed  forces  began  to 
jointly  manage  their  assets.  New  agencies  were  created  while  others  were  combined.  New 
civihan  administrative  roles  and  positions,  formerly  occupied  only  by  mihtary  personnel  were 
created;  mhdng  mihtary  and  civihans  in  the  MoD.  This  changing  scenario  brought  many  new 
needs.  One  newly  recognized  issue  was  the  lack  of  specific  information  systems  to  support  the 
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strategic  decision  process  concerning  weapon  systems  acquisition  within  the  defense 
organizations.  Another  issue  was  the  lack  of  an  Integrated  Logistics  Information  Database 
System  (ILIDS)  that  could  link  each  service  and  its  related  organizations  to  the  MoD  with 
relevant  logistics  information  concerning  the  acquisition  and  support  of  weapon  systems. 

Almost  concurrently  with  these  changes  and  needs,  the  former  Brazihan  Ministry  of 
Aeronautics  (MoA)  -  see  Figure  2  -  started  a  program  in  1993  aiming  to  achieve  an  integration 
of  the  information  systems  within  the  Brazilian  Air  Force  (BAF)  Materiel  Command 
(COMGAP)  -  see  Figure  3.  The  progum,  now  under  responsibihty  of  the  Aeronautical 
Command  (former  MoA),  called  Integrated  System  of  Logistics  Materiel  and  Services 
(SILOMS),  integrates  in  a  single  corporate  database  system  for  all  logistics  information  related 
to  maintenance,  supply,  and  transportation  within  the  COMGAP.  The  overall  goal  of  the 
system  is  to  provide  information  to  support  the  logistics  decision  makers  at  all  three  decision 
levels  within  COMGAP’s  organizations:  operational  (bases,  squadrons  and  depots),  tactical 
(sector  materiel  commands)  and  strategic  (materiel  command).  By  the  end  of  2002  the  system 
is  expected  to  improve  the  capabihty  of  COMGAP’s  organizations  to  control  and  manage 
assets,  including  weapons  systems  and  related  equipment,  as  well  as  track  needs  during  a 
systems’  hfe  cycle.  The  system  will  also  provide  a  clear  vision  of  the  movement  of  materials 
within  the  depots  and  related  bases.  Another  important  feature  of  the  system  is  to  allow  a 
variety  of  queries  in  the  corporate  database  to  collect  statistical  data  that  could  help  the 
measurement  of  key  performance  parameters  related  to  maintenance  activities  as  well  as 
rehabihty  and  avaUabihty  of  the  assets  being  controlled. 
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Figure  2.  Simplified  Brazilian  Aeronautical  Command  Stmcture 


By  the  end  of  2003,  SILOMS  is  supposed  to  fink  one  hundred  and  eighty  organizations 
and  more  than  two  thousands  workstations  in  a  common  network.  With  some  adaptations,  the 
system  has  the  capability  to  fiU  in  the  gap  that  exists  in  the  MoD’s  Logistics  and  Mobilization 
Agency  (SELOM),  by  allowing  integrated  management  of  aU  needs  within  the  military  in 
supporting  their  weapon  systems.  SILOMS  may  be  used,  for  instance,  in  helping  identify  similar 
parts  needed  by  aU  defense  organizations  and  allowing  SELOM  to  employ  a  consolidated 
acquisition  of  supplies,  thereby  promoting  savings  and  improving  the  efficiency  of  the  weapon 
system  acquisition  process  and  their  associated  fife  cycle. 

Brazil  is  rethinking  its  own  government  stmcture  in  fight  of  shrinking  budgets  while  the 
move  to  globalization  is  taking  hold.  New  ways  of  management  and  control  over  government 
activities  and  expenditures  have  to  be  found  to  improve  the  efficiency  of  all  departments  and 
agencies,  while  at  the  same  time  improving  their  activities.  In  this  situation,  SILOMS  has  arisen 
to  be  a  possible  solution  to  some  problems  facing  MoD.  This  research  will  present  a  study 
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about  the  feasibility  of  using  SILOMS  as  a  solution  to  fill  in  the  needs  of  a  logistics  information 


system  for  MoD’s  agency  SELOM. 


Figure  3.  Simphfied  COMGAP  structure  with  subordinated  Depots 


Background 

When  an  information  system  is  developed,  the  analysis  of  the  user’s  needs  lead  to  very 
specific  requirements  that  fit  a  narrow  scope  environment  in  which  the  system  is  supposed  to 
operate.  The  users  of  a  system  in  an  organization  may  be  satisfied  with  the  system’s  features 
that  have  indeed  been  developed  to  fit  their  needs.  On  the  other  hand,  problems  may  appear 
when  trying  to  operate  a  system  in  an  environment  that  was  not  foreseen,  or  where  the  users 
have  not  been  involved  in  the  requirements  analysis  during  the  beginning  developmental  phases 
of  the  system. 
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Even  when  trying  to  adapt  an  existing  information  system  to  a  similar  environment  or 


organization,  some  considerations  related  to  the  new  organizations  can  lead  to  an  unsuccessful 
implementation  or  provoking  strong  users’  opposition  in  using  the  system.  For  instance, 
SILOMS’s  has  experienced  success  since  the  beginning  of  2001  when  it  was  implemented  in 
six  Aeronautical  Depots  Level  Maintenance  Center  (PAMA),  one  Electronic  Depot  Level 
Maintenance  Centers  (PAME),  and  related  air  force  bases  linked  with  an  integrated  database 
system.  The  success  of  the  implementation  was  only  possible  after  solving  many  problems  that 
had  come  up  when  organizations  were  being  analyzed  in  an  effort  to  get  the  overall  picture  of 
the  COMGAP  logistics  activities.  Constraints  such  as,  organizational  and  cultural  differences, 
pohtical,  resources  and  environmental  issues  as  well  as  internal  processes  related  to  logistics 
showed  to  be  a  challenge  that  faced  the  analysts  even  in  similar  COMGAP’ s  sectors.  An 
extensive  study  about  the  way  to  perform  tasks  in  the  organizations  took  place  to  allow 
standardization  of  processes;  and,  at  the  same  time,  meeting  users’  needs.  The  SILOMS 
program  brought  attention  to  the  fact  that  even  in  the  BAF  materiel  command  sectors  that  were 
supposed  to  perform  tasks  with  similar  processes,  that  was  not  always  the  case.  The  degree  of 
standardization  of  process  within  COMGAP’ s  sectors  was  a  key  issue  to  the  success  or  failure 
of  the  SILOMS  program.  Fortunately,  a  considerable  degree  of  standardization  has  been 
achieved  within  the  COMGAP  agencies,  after  considerable  efforts  to  perform  changes  that 
were  directed  to  aU  levels  of  management  within  the  materiel  command. 

Since  1993  when  SILOMS  started  to  be  implemented,  many  problems  such  as  cultural 
differences  and  different  ways  in  performing  activities  have  challenged  the  analysts  and  also  the 
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COMGAP’s  administration  in  trying  to  standardize  the  processes  and  tasks  related  to  logistics 


support  within  the  materiel  command  sectors.  The  success  of  SILOM’s  implementation  was  in 
great  part  due  to  the  successfully  standardization  of  the  processes  and  activities  that  took  place 
in  all  development  phases  of  the  integrated  information  system.  By  the  end  of  2003,  SILOMS 
is  expected  to  link  and  support  the  operation  of  one  hundred  and  eighty  organizations  and  more 
than  two  thousand  workstations  in  a  common  network  of  the  BAF. 

Problem  Statement 

In  order  to  handle  its  obhgations,  the  new  MoD’s  agency  SELOM  will  need  an 
information  system  capable  of  managing  logistics  data  and  information  from  ah  rruhtary  services 
and  security  forces. 

Considering  the  problems  described  in  the  previous  sections,  we  can  expect  that 
SELOM  (that  needs  an  information  system  to  meet  the  needs  not  only  of  BAF,  but  also  the 
Brazihan  Army  (EB)  and  Brazihan  Navy  (MB))  wiU  face  challenges  in  adopting  a  new  integrated 
information  system  -  even  if  it  chooses  to  adopt  SILOMS.  To  develop  an  information  system 
with  a  corporate  database,  the  degree  of  standardizations  within  the  organization  can  influence 
the  success  of  such  implementation.  It  is  expected  that  a  careful  study  of  the  differences  related 
to  logistics  in  the  services  within  the  MoD  take  place  before  starting  the  implementation  of  such 
system.  Also,  if  SELOM  decides  to  use  SILOMS,  a  study  about  the  constraints  such  as 
organizational  and  cultural  differences,  pohtical,  resources  and  environmental  issues  is  needed. 
Otherwise,  the  same  problems  that  challenged  the  SILOMS  implementation  within  the  BAF  are 
expected  to  occur  when  attempting  to  use  it  as  a  base  system  to  the  MoD’s  agency  SELOM. 
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The  implementation  of  an  integrated  information  system  has  inherent  ehaUenges. 


Differences  in  organizations  and  cultures,  political,  resources  and  environmental  issues  or  in  the 
way  tasks  are  performed,  are  key  issues  to  be  observed  in  attempting  to  do  so.  The  same  is 
applicable  when  trying  to  adapt  an  already  existing  system  to  fill  in  the  need  of  another 
organization.  In  such  new  environment,  a  key  issue  is  to  assess  the  feasibility  of  proceeding  with 
an  adaptation  of  an  existing  information  system  or  if  it  is  better  to  build  a  completely  new 
system.  If  SELOMS  chooses  to  use  the  SILOMS,  what  constraints  exists  that  can  threat  the 
success  of  its  implementation  in  the  MoD? 

Research  Objectives  and  Questions 

The  objective  of  this  research  is  to  provide  a  method  to  measure  the  effort  and 
feasibility  of  using  SILOM’s  functions  in  the  SELOM’s  environment.  In  this  way,  it  will 
contribute  to  the  integration  of  the  logistics  management  and  the  expected  benefits  that  such  a 
system  can  provide  for  the  Brazilian  MoD’s  agency.  The  research  is  undertaken  to  answer 
research  questions  about  the  feasibility  of  adopting  the  SILOMS  as  a  base  logistics  information 
system,  or  whether  it  is  better  to  start  the  implementation  of  a  completely  new  system. 

Research  Questions 

The  implementation  of  an  integrated  information  system  has  inherent  challenges  as 
discussed  in  previous  sections.  SELOM  has  decided  to  rely  on  a  logistics  information  system  to 
better  perform  its  activities.  Is  it  feasible  to  use  SILOMS  as  a  baseline  system  to  manage 
logistics  needs  and  assets  within  the  armed  forces  and  MoD? 
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To  assess  the  feasibility  of  doing  so  requires  a  study  and  a  methodology  to  determine 


the  constraints  suitabihty  of  SILOMS’s  functions  to  fulfill  other  services  needs.  By  doing  so,  the 
research  wiU  offer  SELOM  a  tool  to  support  the  decision  of  whether  it  is  better  to  develop  a 
new  system  or  whether  take  the  advantage  of  using  an  existing  one.  How  to  assess  the 
feasibihty  and  risks  of  the  implementation  of  SILOMS  in  the  MoD? 

Investigative  Questions 

A  top-down  approach  helps  to  answer  the  research  questions  in  the  way  that  allows 
breaking  the  research  questions  into  more  specific  questions  to  facihtate  the  analysis.  Specific 
questions  have  to  be  answered  in  order  to  assess  the  feasibility  of  using  SILOMS  in  the  MoD 
environment: 

•  What  are  the  factors  critical  to  the  successful  implementation  of  SILOMS  in  the 
MoD? 

•  What  is  an  appropriate  method  available  to  assess  or  predict  risks  involved  in  the 
implementation  of  SILOMS  in  the  MoD? 

•  How  would  we  quantify  the  degjiee  of  risks  in  order  to  help  the  decision  making 
process  of  adopting  SILOMS  in  the  MoD? 

•  Can  a  probabihty  of  success  be  obtained  from  this  methodology? 

By  answering  these  questions  the  research  wiU  consohdate  information  to  serve  as  an 
input  to  the  decision  makers  for  assessing  the  feasibihty  of  using  SILOMS  as  a  basehne  system 
to  SLLOM  agency. 
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Data  Sources  and  Analysis 


To  answer  the  research  and  investigative  questions  it  is  necessary  to  understand  the 
constraints  that  apply  when  implementing  a  new  information  system  in  organizations.  The 
objective  and  subjective  data  of  the  constraints  will  be  gathered  from  personal  interviews  and 
questionnaires  submitted  to  key  personnel  in  MoD’s  agencies  and  systems  analysts  from  the 
Integrated  System  of  Logistics  Materiel  and  Services  Task  Force  (GT-SILOMS).  The 
interviews  will  be  based  on  a  questionnaire  proposed  originally  by  a  technical  report  of  The 
Carnegie- Mellon  University’s  Software  Engineering  Institute  and  modified  by  the  researcher  to 
fit  the  specific  circumstances  that  apply  to  this  research.  An  evaluation  about  the  extent  and  the 
feasibility  of  applying  SILOMS  to  other  armed  forces  and  MoD  can  then  be  performed.  The 
data  available  will  then  be  tabulated  using  a  rating  scale  and  rank  order  procedure,  by 
relevance,  for  the  questions  asked  for  the  interviewees.  In  this  way  the  research  will  provide  a 
way  to  quantify  a  typical  quahtative  assessment  in  order  to  better  help  the  decision  makers  to 
evaluate  the  implementation  of  SILOMS  in  the  MoD. 

Scope/Limitations 

In  a  study  to  assess  the  feasibihty  of  using  an  existing  information  system  in  different 
organizational  environments  it  is  necessary  to  understand  each  of  the  of  the  constraints  that 
affects  the  implementation  of  the  system,  and  also  the  specific  needs  of  the  new  organization 
where  the  system  is  supposed  to  operate.  In  such  a  study,  the  data  tables,  functions  and  internal 
routines  of  the  system  have  to  be  analyzed  and  a  test  of  fitness  to  the  new  environment  has  to  be 
performed.  SILOMS  has  more  than  1,400  functions  subdivided  under  six  major  logistics 
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functions:  personnel,  facilities,  supply,  maintenance,  transportation  and  independent.  These 
major  functions  enclose  other  functions  modules  as  seen  in  the  Function  Break-down  Stiucture 
(FBS)  in  Figure  4. 


Figure  4.  Simphfied  SILOMS's  FBS  -  Logistics  functions.  COMGAP's  Taxonomy 


This  study  assumes  that  it  is  technically  possible  to  use  and/or  adapt  SILOMS’s 
functions  to  satisfy  the  MoD’s  needs.  Then,  factors  as  contracting,  resources,  program 
interfaces  constraints,  or  other  constrains  of  this  nature,  will  not  be  explored.  The  focus  will  be 
on  the  organization,  cultural  and  pohtical  aspects  that  can  threat  the  successful  implementation  of 
SILOMS  in  the  MoD’s  agency  SELOM.  These  aspects  will  only  be  explored  within  middle 
and  high-level  managers  of  MoD’s  agencies.  The  method  proposed  in  this  study  could  be  used 
to  a  more  complete  assessment  enclosing  other  factors,  constraints  and  personnel  in  future 
studies. 
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Chapter  Summary 


This  chapter  presented  some  issues  that  Brazil’s  MoD  agency  SELOM  is  facing  in  the 
integration  of  the  armed  forces  logistics  information  system.  BAF’s  logistics  information  system, 
SILOMS,  has  been  described  as  a  potential  system  that  can  fulfill  SELOM’S  needs.  Problems 
that  arise  in  developing  or  adapting  an  information  system  were  described  and  some  inherent 
challenges  were  highhghted.  Questions  that  have  to  be  answered  to  assess  the  feasibihty  of 
using  SIEOMS  as  a  baseline  system  for  the  MoD  environment  were  described.  Data  source 
and  analysis  were  presented.  Finally  the  scope  of  the  study  was  presented,  limiting  the  study  to 
the  organization,  cultural  and  pohtical  factors  and  constraints  that  could  threat  the  successful 
implementation  of  SIEOMS  in  the  MoD. 

In  Chapter  II  a  hterature  review  about  the  methods  and  concepts  used  to  perform  this 
study  are  presented.  Chapter  ni  will  present  the  selected  methodology  to  perform  the  study, 
while  Chapter  IV  will  present  the  data  obtained  and  the  required  analysis  to  assess  the  feasibihty 
of  the  implementation  of  SIEOMS  in  the  MoD’s  agency  SEEOM.  The  last  chapter  wiU  present 
the  conclusions  and  recommendations  when  apphcable. 


12 


II.  Literature  Review 


Introduction 

Before  assessing  the  feasibility  of  using  SILOMS  in  the  MoD’s  Agency,  it  is  necessary 
to  gather  relevant  information  concerning  the  methods  available  in  making  interviews  and  also 
about  risk  assessment  and  identification  methods.  Also,  a  hterature  review  about  project 
management  and  inherent  risks  associated  is  required  since  the  nature  of  the  work  might  fit  the 
definition  of  a  project. 

Then,  the  hterature  review  wiU  first  explore  the  definitions  and  characteristics  of  a 
quahtative  research  as  weU  as  considerations  about  surveys  and  the  use  of  questionnaires, 
interviews  and  Likert’s  scale,  that  will  apply  to  the  present  study.  Second,  a  literature  review  of 
the  relationship  between  projects  and  risks,  as  well  as  the  appropriateness  of  using  project 
management  approach  in  activities  with  inherent  risks  associated  wiU  be  discussed.  Third,  the 
chapter  will  discuss  and  present  the  concepts  of  managing  risks  in  software  development  project 
and  available  risk  analysis  methods.  Fourth,  the  Taxonomy- Based  Risk  Identification  Method 
(TRI),  a  method  to  risk  identification  in  software  development  environments,  and  the  derived 
Taxonomy-Based  Questionnaire  (TBQ)  wiU  be  presented.  FinaUy,  a  chapter’s  summary  will 
briefly  list  the  concepts  and  theories  discussed  in  Chapter  II. 
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Qualitative  Research 


This  section  presents  the  definition  and  characteristics  of  a  qualitative  research  and  the 

relationship  with  this  study.  Then,  the  section  will  present  the  situations  where  the  qualitative 

research  study  may  apply  and  its  five  common  types.  The  last  part  will  present  the 

characteristics  of  surveys,  related  questionnaires  and  interviews,  and  the  Likert’s  scale. 

Since  the  focus  of  this  study  are  the  organization,  cultural  and  political  aspects  that  can 

threat  the  successful  implementation  of  SILOMS  in  the  MoD’s  agency  SELOM,  one  can 

expect  that  many  dimensions  and  subjective  aspects  may  appear  when  performing  the  study.  A 

qualitative  study  and  its  methods  have  the  advantage  of  going  in  depth  in  the  problem  by 

allowing  more  flexibility  to  the  researcher  in  a  real  world  environment  or  natural  setting  with 

inherent  subjective  aspects.  Then,  when  subjective  aspects  are  present,  a  qualitative  research  is 

best  recommended.  And  in  order  to  get  a  complete  understanding  of  the  constraints  that  may 

occur  in  the  implementation  of  SILOMS  in  MoD,  a  qualitative  research  and  its  methodologies 

has  to  be  employed.  As  stated  in  Leedy: 

The  term  qualitative  research  encompasses  several  approaches  to  research 
that  are,  in  some  respects,  quite  different  from  one  another.  First,  they  focus  on 
phenomena  that  occur  in  natural  settings  -  that  is,  in  the  “real  world.”  And 
second  they  involve  studying  those  phenomena  in  all  their  complexity.  Qualitative 
researchers  rarely  try  to  simplify  what  they  observe.  Instead,  they  recognize  that 
the  issue  they  are  studying  has  many  dimensions  and  layers,  and  so  they  try  to 
portray  the  issue  in  its  multi-faceted  form.  (7 : 147) 

Different  from  quantitative  research,  which  is  more  appropriate  for  studying  physical 
events,  the  qualitative  research  is  more  adequate  to  explore  human  events  that  have  inherent 
subjectivism  and  where  multiple  perspectives  can  be  held  by  different  individuals.  These 
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multiple  perspective,  in  turn,  may  have  equal  validity  or  add  value  to  the  data  analysis  and 

conclusions.  Furthermore,  these  multiple  perspectives  and  revealing  their  natures  end  up  of 

being  one  important  goal  of  quahtative  studies. 

Furthermore,  many  qualitative  researchers  beheve  that  there  isn’t  necessarily  a 
single,  ultimate  Tmth  to  be  discovered.  Instead,  there  may  be  multiple 
perspectives  held  by  different  individual,  with  each  of  these  perspectives  having 
equal  validity,  or  tmth  (CresweU,  1998;  Guba  &  Lincoln,  1988).  One  goal  of  a 
quahty  study,  then,  might  be  to  reveal  the  nature  of  these  multiple  perspectives. 

(7:147) 

In  quahtative  studies,  more  often  the  researcher  formulates  only  general  problems  and 

asks  general  questions  about  a  phenomenon  he  is  studying.  This,  although,  doesn’t  mean  that 

the  problems  and  questions  remain  vague.  As  the  researcher  proceeds  with  the  study,  the 

nature  of  the  phenomenon  being  studied  becomes  more  understandable  and  the  researcher 

becomes  better  able  to  ask  specific  questions. 

These  research  problems  and  questions  do  not  remain  so  loosely  defined, 
however.  As  a  study  proceeds,  the  quahtative  researcher  gets  an  increasingly 
better  handle  on  the  nature  of  the  phenomenon  under  investigation  and  so 
becomes  increasingly  better  able  to  ask  specific  questions.  (7 : 148) 

In  general,  quahtative  studies  do  not  help  the  researcher  to  identify  cause- and- effect 

relationships  to  answer  questions  about  whether  a  cause  or  some  specific  circumstance  has 

provoked  the  effect.  If  such  answers  are  needed,  the  quantitative  approach  is  needed.  The 

decision  when  to  choose  to  proceed  with  a  qualitative  approach  depends  upon  the  purpose  of 

the  study.  Typicahy  quahtative  studies  serve  one  or  more  of  the  fohowing  purposes,  according 

Peshkin  reported  by  Leedy  in  Practical  Research  -  Planning  and  Design  (7 : 148): 

•  Description.  They  can  reveal  the  nature  of  certain  situations,  settings, 
processes,  relationships,  systems,  or  people. 
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•  Interpretation.  They  enable  a  researcher  to  (a)  gain  insights  about  the 
nature  of  a  particular  phenomenon,  (b)  develop  new  concepts  or 
theoretical  perspectives  about  the  phenomenon,  and/or  (c)  discover  the 
problems  that  exist  within  the  phenomenon. 

•  Verification.  They  allow  a  researcher  to  test  the  vahdity  of  certain 
assumptions,  claims,  theories,  or  generahzations  within  real-world 
contexts. 

•  Evaluation.  They  provide  a  means  through  which  a  researcher  can 
judge  the  effectiveness  of  particular  pohcies,  practices,  or  innovations. 

(7:148) 

Quahtative  research  studies  may  be  performed  using  some  common  designs,  that  is, 
case  studies,  ethnographies,  phenomenological  studies,  grounded  theory  studies,  and  content 
analyses.  These  types  of  quahtative  studies  are  briefly  presented  below,  summarized  from 
Leedy  (7:149-157). 

•  Case  Study.  A  particular  individual,  program,  or  event  is  studied  in  depth  for  a 
defined  period  of  time.  They  are  common  in  medicine,  education,  pohtical 
science,  law,  psychology,  sociology,  and  anthropology. 

•  Ethnography.  Different  from  the  case  study,  ethnography  looks  in  depth  at  an 
entire  group  that  shares  a  common  culture.  The  focus  is  on  the  everyday 
behaviors  of  the  people  in  the  group  in  order  to  identify  cultural  patterns. 

•  Phenomenological  Study.  It  is  a  study  that  attempts  to  understand  people’s 
perceptions,  perspectives,  and  understandings  of  a  particular  situation. 

•  Grounded  Theory  Study.  Uses  a  prescribed  set  of  procedures  for  analyzing 
data  and  constmcting  a  theoretical  model  from  them.  Has  its  roots  in  sociology 
but  is  now  used  in  anthropology,  education,  nursing,  psychology,  and  social 
work. 

•  Content  Analysis.  It  is  a  detailed  and  systematic  examination  of  the  contents 
of  a  particular  body  of  material  for  the  purpose  of  identifying  patterns,  themes, 
or  biases.  Typicahy  performed  over  data  found  in  human  communication 
forms,  as  books,  newspapers,  films,  television,  art,  music,  videotapes  etc. 
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In  the  attempt  to  define  which  type  of  design  apphes  to  a  specific  study  is  important  to 


distinguish  characteristics  of  the  different  quahtative  designs.  Then,  a  decision  can  be  made 
about  the  most  appropriate  for  the  purpose  of  the  study.  Table  1 ,  extracted  from  Leedy 
(7:157)  shows  the  characteristics  of  each  type  of  quahtative  design. 

Table  1 .  Distinguishing  Characteristics  of  Different  Quahtative  Designs  (7 : 157) 


Design 

Purpose 

Focus 

Methods  of  Data  Collection 

Methods  of  Data  Analysis 

Case  Study 

To  understand  one 
person  or  situation 
(or  perhaps  a  very 
small  number)  in 
great  depth 

One  case  or  few  cases 

within  its/their 
natural  setting 

Observations 

Interviews 

Appropriate  written 
documents  and/or  audiovisual 

material 

Categorization  and 
interpretation  of  data  in 
terms  of  common 

themes 

Synthesis  into  an  overall 
portrait  of  the  cases 

Ethnography 

To  understand  how 

behaviors  reflect 

the  culture  of 

group 

A  specific  field  site  in 
which  a  group  of 
people  share  a 
common  culture 

Participant  observations 
Structured  or  unstructured 

interviews  with  “informants” 

Artifact/document  collection 

Focus  on  significant 

events 

Phenomenological 

To  understand  an 

A  particular 

In  depth  unstructured 

Search  for  meaning  units 

study 

experience  from  the 
participants  point 
of  view 

phenomenon  as  it  is 
typically  lived  and 
perceived  by  humans 

interviews 

Purposeful  sampling  of  5-25 
individuals 

that  reflect  various 
aspects  of  the 
experience 

Integration  of  meaning 
units  into  a  typical 
experience 

Grounded  theory 

To  derive  a  theory 

Human  actions  and 

Interviews 

Prescribed  and 

study 

from  data  collected 
in  a  natural  setting 

interactions,  and  how 
they  result  from  and 
influence  one  another 

Any  other  relevant  data 

sources 

systematic  method  of 
coding  the  data  into 
categories  and 
identifying 

Continual  interweaving 
of  data  collection  and 
data  analysis 

Construction  of  a  theory 
from  the  categories  and 
interrelationships 

Content  analysis 

To  identify  the 
specific 

characteristics  of  a 
body  of  material 

Any  verbal,  visual,  or 
behavioral  form  of 

communication 

Identification  and  possible 
sampling  of  the  specific 
material  to  be  analyzed 
Coding  of  the  material  in 
terms  of  predetermined  and 
precisely  defined 
characteristics 

Tabulation  of  the 
frequency  of  each 
characteristic 

Descriptive  or 
inferential  statistical 
analyses  as  needed  to 
answer  the  research 
question 
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Surveys  -  Questionnaires,  Interviews  and  Likert  Scales 


This  section  will  present  definitions  and  characteristics  of  a  qualitative  research  and  the 
relationship  with  this  study.  Then,  the  section  will  present  the  situations  where  qualitative 
research  study  may  apply  and  its  five  common  types.  The  last  part  will  present  the 
characteristics  of  surveys,  related  questionnaires,  interviews,  and  finally  the  two  techniques  that 
allow  the  evaluation  and  quantification  of  peoples’  perceptions;  checklist  and  Likert  scales. 

Surveys  are  used  in  order  to  collect  data  in  many  areas  of  research.  The  data  gathered 
by  surveys  turns  into  important  information  to  those  in  head  of  organizations,  either  government 
or  private  owned  companies. 

Surveys  are  used  today  to  collect  data  on  almost  every  conceivable  subject, 
including  attitudes  about  presidential  candidates,  television  viewing  habits  or  the 
health  and  well-being  of  the  populace.  (15:1) 

Surveys  are  present  on  everyday  business  nowadays,  either  by  questionnaires  or 
interviews.  The  questionnaires  are  a  kind  of  survey  that  can  be  “self- administered”  and  usually 
can  be  sent  by  mail.  Interviews  are  another  way  to  take  surveys  and  usually  require  more  time 
to  be  done  -  frequently  named  as  interview-based  surveys  (15)  -  due  to  the  fact  that  normally 
requires  a  face-to-face  contact  between  the  interviewer  and  the  interviewed. 

Also,  interviews  “can  yield  a  great  deal  of  useful  information.”(7 : 159),  where  the 
researcher  can  ask  questions  related  to  facts,  people  behefs,  feehngs,  motives,  standards  for 
behavior,  etc.  According  Leedy,  when  interviews  are  apphed  to  quahtative  study  they  have 
some  particular  characteristics: 
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The  interviews  in  a  qualitative  study  are  rarely  as  strue  hired  as  the  interviews 
condueted  in  a  quantitative  study  (...)•  Instead,  they  are  either  open-ended  or 
semi- structured,  in  the  latter  case  revolving  around  a  few  central  questions. 
Unstructured  interviews  are,  of  course,  more  flexible  and  more  likely  to  yield 
information  that  the  researcher  hadn’t  planned  to  ask  for;  (7 : 159) 


Surveys  are  often  used  to  learn  about  people’s  perceptions  and  opinions,  and  since 

behaviors  and  attitudes  are  complex  and  difficult  to  evaluate  or  quantify,  there  exist  two 

techniques  that  allows  the  evaluation  and  quantification  in  such  cases.  They  are  checklist  and 

rating  scales.  The  first  one  is  defined  in  Leedy  as: 

A  checldist  is  a  hst  of  behaviors,  characteristics,  or  other  entities  that  a  research 
is  looking  for.  Either  the  researcher  of  the  survey  participant  (depending  on  the 
study)  simply  checks  whether  each  item  on  the  hst  is  observed,  present,  or  hue; 
or  else  not  observed,  present,  or  true.  (7 : 197) 

The  rating  scale  is  a  technique  that  ahows  the  researcher  to  assign  to  a  parameter  of 

interest  some  sort  of  a  continuum  range  of  values  that  can  be  further  quantified  in  numerical 

terms.  Rating  scales  were  first  developed  and  reported  by  Rensis  Likert  (9)  and  are  known  as 

Likert  scales.  Leedy  describe  rating  scales  as  being: 

(...)  more  useful  when  a  behavior,  attitude,  or  other  phenomenon  of  interest 
needs  to  be  evaluated  on  a  continuum  of,  say,  “inadequate”  to  “exceUent,” 

“never”  to  “always,”  or  “strongly  disapprove”  to  “strongly  approve.”  (7:197) 


Also  in  as  reported  in  Surveys  with  Confidence:  A  Practical  Guide  to  Survey  Research 
Using  SPSS,  Likert  scales  are: 

(...)  a  ranked  hst  of  responses  that  mns  form  one  to  another  (Strongly  disagree 
to  Strongly  agree).  The  psychologist  Rensis  Likert  was  the  first  to  study  these 
scales  in  some  depth,  thus  they  are  referred  to  as  Likert  scales.  (15:15) 
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This  section  has  presented  the  definition  and  characteristics  of  a  qualitative  research  and 


the  relationship  with  this  study.  Then,  the  section  has  presented  the  situations  where  quahtative 
research  study  may  apply.  Also,  the  five  common  types  of  qualitative  research  and  their  most 
common  uses  were  presented.  The  last  part  has  presented  the  characteristics  of  surveys, 
related  questionnaires,  interviews,  and  finally  the  two  techniques  that  allow  the  evaluation  and 
quantification  of  peoples’  perceptions;  checklist  and  Likert  scales. 

Projects  and  Risks 

In  the  attempt  to  adopt  SILOMS  in  MoD  one  can  expect  that  this  is  going  to  be  a 
challenging  effort  given  that  the  activities  to  be  performed  in  the  attempt  will  be  unique  and 
unfamiliar.  The  outcome  of  such  an  attempt  might  be  surrounded  by  uncertainties.  Then,  it  will 
involve  risk  of  failure  in  this  effort  concerning  the  feasibihty  and  the  results  associated  in  case  of 
the  outcome  do  not  be  the  expected  by  the  MoD  users.  The  existences  of  such  characteristics 
are  some  of  those  that  define  an  activity  as  a  project.  A  project  management  approach  will, 
likely,  be  the  preferred  choice  to  handle  the  implementation  of  SILOMS  in  the  MoD.  Then,  a 
hterature  review  about  project’s  characteristics  and  management,  as  well  as  the  inherent  risks 
involved  is  justified.  These  concepts  will  help  in  the  definition  of  the  method  used  to  assess  the 
feasibility  of  using  SILOMS  in  the  MoD. 

Projects  and  Project  Management  Approach 

In  the  attempt  to  accomphsh  a  goal,  an  organization  may  face  unique  circumstances 
surrounding  the  tasks  to  be  performed.  The  famihaiity  with  the  tasks  and  the  acknowledgement 
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of  the  processes  to  be  performed  in  addition  to  weU  defined  statement  and  requirements  of  the 


end- item  or  product  constitute  key  features  that  whl  define  the  success  of  the  activity.  Such 
activities  with  certain  characteristics  can  be  called  as  project.  In  A  Guide  to  Project 
Management  Body  of  Knowledge,  reported  by  Nicholas  in  (10:4),  project  is  defined  as: 

A  project  can  thus  be  defined  in  terms  of  its  distinctive  characteristics  -  a  project 

is  a  temporary  endeavor  undertaken  to  create  a  unique  product  or  service. 

(10:4) 

According  John  M.  Nicholas  in  Project  Management  for  Business  and  Technology, 
some  characteristics  can  be  used  to  warrant  classifying  an  activity  as  a  project: 

•  A  project  involves  a  single,  definable  purpose,  end-item,  or  result,  usually 
specified  in  terms  or  cost,  schedule,  and  performance  requirements. 

•  Every  project  is  unique  in  that  it  requires  doing  something  different  than  was 
done  previously.  Even  in  “routine”  projects  such  as  home  constmction, 
variables  such  as  terrain,  access,  zoning  laws,  labor  market,  pubhc  services, 
and  local  utilities  make  each  project  different.  A  project  is  a  one-time 
activity,  never  to  be  exactly  repeated  again. 

•  Projects  are  temporary  activities.  And  ad  hoc  organization  of  personnel, 
material,  and  facilities  is  assembled  to  accomphsh  a  goal,  usually  within  a 
scheduled  time  frame;  once  the  goal  is  achieved,  the  organization  is 
disbanded  or  reconfigured  to  begin  the  work  on  a  new  goal. 

•  Projects  cut  across  organizational  lines  because  they  need  the  skills  and 
talents  from  multiple  professions  and  organizations.  Project  complexity  often 
arises  form  the  complexity  of  advanced  technology,  which  creates  task 
interdependencies  that  may  introduce  new  and  unique  problems. 

•  Given  that  a  project  differs  from  what  was  previously  done,  it  also  involves 
unfamiliarity.  It  may  encompass  new  technology  and,  for  the  organization 
undertaken  the  project,  posses  significant  elements  of  uncertainty  and  risk. 

•  The  organization  usually  has  something  at  stake  when  doing  a  project.  The 
activity  may  call  for  special  scmtiny  or  effort  because  of  failure  would 
jeopardize  the  organization  or  its  goals. 
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•  Finally,  a  project  is  the  process  of  working  to  achieve  a  goal;  during  the 
process,  projects  pass  through  several  distinct  phases,  called  the  project  life 
cycle.  The  tasks,  people,  organizations,  and  other  resources  change  as  the 
project  moves  from  one  phase  to  the  next.  The  organization  stmcture  and 
resource  expenditures  slowly  build  with  each  succeeding  phase;  peak;  and 
then  dechne  as  the  project  nears  completion.  (10:4) 

The  above  characteristics  when  found  in  an  activity  may  lead  to  the  adoption  of  a 

particular  approach  of  management  called  project  management  as  defined  in  A  Guide  to  Project 

Management  Body  of  Knowledge,  reported  by  Nicholas: 

Project  Management  is  the  apphcation  of  knowledge,  skills,  tools,  and 
techniques  to  project  activities  in  order  to  meet  or  exceed  stakeholder  needs  and 
expectations  from  a  project.  (10:4) 

Some  key  characteristics  of  project  management  are  summarized  from  Nicholas 
(10:22-23): 


•  One  person  -  the  project  manager  -  heads  the  project  organization 

•  Project  manager  unifies  all  efforts  to  achieve  project  objectives 

•  Several  functional  areas  often  perform  the  work 

•  Project  manager  is  responsible  for  integrating  the  efforts  of  the  functional  areas 
working  on  the  project 

•  Project  manager  negotiates  directly  with  functional  managers  for  support 

•  Project  focuses  on  dehvering  a  particular  product  or  serwce  at  a  certain  time  and 
cost  and  to  the  satisfaction  of  technical  requirement. 

The  program  manager  has  the  key  role  of  seeing  the  “big  picture”  or  taking  a  system 

approach  to  the  project  to  assure  that  each  task  being  performed  is  in  accordance  to  the  main 

goal  in  the  project.  Ultimately  he  is  the  responsible  to  minimize  the  inherent  risks  involved  in 
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projects  by  quantifying  them  and  by  taking  appropriate  measures  to  avoid  or  minimize  their 
impacts  in  the  project  goal.  The  successful  project  management  rehes  upon  the  need  to 
accomphsh  the  called  “triple  constraint”: 

Every  project  is  constrained  in  different  ways  by  its  scope,  time,  and  costs  goals 

-  the  Triple  Constraint.  (...)  Successful  project  management  means  meeting  aU 

three  goals  -  and  satisfying  the  project  sponsor.  (10:20) 

Project  and  Inherent  Risks 

In  the  attempt  to  accomplish  a  project  goal,  the  project  manager  has  to  assess  the  risks 
involved  in  the  most  difficult  tasks  or  those  surrounded  by  unfamiharity.  The  assessment  and  a 
measure  of  the  risk  involved  in  projects  is  a  practice  that  has  to  be  used  before  starting  a 
project.  One  organization  may  decide  if  it  is  worthy  to  take  the  risk  involved  in  a  specific 
project  only  if  the  organization  can  measure  the  risk  and  the  consequences  of  doing  or  not  doing 
the  project. 

When  an  organization  is  developing  a  computer-based  information  system  or  even  trying 
to  adopt  an  existing  one,  the  characteristics  of  the  activities  and  tasks  to  be  performed  may  fit  aU 
those  described  to  define  a  project.  Software  and  database  development  normally  is  unique 
efforts  to  meet  specific  needs  of  organizations;  involves  many  sectors  or  “cut  across  the 
organizational  lines”  and  is  surrounded  by  unfamiharity,  uncertainty  and  possess  significant 
elements  of  risks. 

In  this  section  we’ve  seen  the  concepts  of  projects  as  weU  as  project  management 
approach  and  the  inherent  risks  involved  in  such  kind  of  activities.  The  adoption  of  SILOMS  in 
the  MoD  can  be  seen  as  an  activity  that  has  all  to  do  with  a  project  and  project  management 
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practices  and  this  allows  the  study  to  apply  some  of  the  project  concepts  to  assess  risks  and 


then  help  the  decision  makers  to  decide  whether  or  not  is  feasible  to  adopt  SILOMS  in  the 
MoD. 

Managing  Risks  and  Software  Project  Risk  Analysis  Methods 

This  section  will  present  the  concepts  of  managing  risks  related  to  projects  and  their 

relation  with  this  study.  Also,  a  brief  discussion  about  risk  analysis  methods  will  be  presented. 

hr  general  risks  arises  when  there  exist  uncertainties,  which  in  turn  is  related  with 

unfamiharity  or  uniqueness  of  an  activity  or  project.  The  experience  of  the  project  team  also 

counts  on  the  possible  risks  involved.  When  both  conditions  exist,  uniqueness  and 

inexperienced  team,  the  outcomes  of  a  project  becomes  more  uncertain  making  it  difficult  to 

know  what  could  go  wrong  and  how  to  avoid  problems  since  the  outcomes  can  be  influenced 

by  factors  that  are  new,  emerging,  or  beyond  manager’s  control.  Stated  by  Nicholas: 

Every  project  is  risky,  meaning  there  is  a  chance  things  won’t  turn  out  exactly  as 
planned.  Project  outcomes  are  determined  by  many  things,  some  that  are 
unpredictable  and  over  which  project  managers  has  httle  control.  (10:336) 

The  notion  of  project  risk  involves  the  concepts  of  the  likelihood  that  some 

problematical  event  will  occur  and  the  impact  if  the  event  does  occur.  And  Nicholas  stated  it  as 

a  join  function  in  the  following  formulation  “Risk  =  /(likelihood,  impact)”  (10:307). 

No  matter  if  only  one  exists,  that  is,  either  the  likelihood  or  the  impact,  the  project  may 

be  considered  risky  whenever  some  particular  outcomes  have  the  probability  of  existing,  such  as 

human  casualties  or  huge  material  losses.  According  Nicholas: 
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One  project  will  be  considered  risky  where  the  potential  impact  is  human  fatahty 
or  massive  financial  loss  even  when  the  likelihood  of  either  is  small.  (10:307) 

The  risk  involved  in  the  case  of  engaging  in  a  project  to  use  SILOMS  in  the  MoD 

might/may  not  be  fatal,  but  certainly  will  involve  financial  losses  if  the  outcome  is  not  the 

expected  one.  Also,  depending  on  the  circumstances,  the  MoD  may  incur  in  a  high  risk  of  not 

having  such  a  system  to  support  a  quick  response  to  the  logistical  support  requirements  in  the 

case  of  the  raise  of  a  conflict.  In  that  case,  the  fatalities  may  occur  due  to  the  fact  that  a  good 

information  system  could  better  help  the  logistics  support  for  a  conflict. 

Risks  are  inherent  to  projects  and  the  consequences  of  failures  may  be  disastrous 

depending  to  the  circumstances.  Before  accepting  the  risk  of  engaging  in  a  project,  the  decision 

makers  have  to  be  able  to  measure  it,  and  then,  decide  if  it  is  worth  to  take  it. 

A  risk  analysis  related  to  software  development  projects  can  be  defined  as  the 

evaluation  of  the  risk  potentials  associated  with  the  development  process  and  also  those  risks 

associated  with  the  tools,  methods  and  approaches  to  be  used  during  the  software  project 

development.  The  inadequate  software  project  risk  analysis  is  associated  with  many  factors  and 

may  cause  the  failure  of  a  project.  Jones  defines  “Inadequate  Software  Project  Risk  Analysis” 

as: 

A)  Failure  to  consider  or  properly  evaluate  the  risk  potential  of  significant  software 
projects  prior  to  commencement; 

B)  Failure  to  consider  or  properly  evaluate  the  risk  potentials  of  significant  software 
projects  based  on  changes  after  development  begins; 

C)  Failure  to  consider  risks  associated  with  tools,  methods,  and  approaches  prior  to 
acquisition  and  deployment.  (6:254) 
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Jones  (6)  considers  the  roots  of  inadequate  risk  analysis  due  to  the  fact  that  risk  analysis 


is  taught  neither  by  software  engineering  curricula  nor  by  enterprise  training  curricula.  Also  he 
highhghts  the  fact  that  “serious  risk  analysis  is  a  fairly  recent  phenomenon”  (6:255)  and  that  due 
to  corporate  culture  of  the  enterprises,  they  tends  to  ignore  risk- related  conditions. 

This  section  presented  the  concepts  of  managing  risks  related  to  projects  as  well  as  the 
relation  of  this  concept  with  this  study.  Also,  a  brief  discussion  about  risk  analysis  methods  has 
been  presented. 

Taxonomy-Based  Risk  Identification  Method 

This  subsection  will  present  the  Taxonomy- Based  Risk  Identification  Method  (TRI), 
proposed  by  a  report  of  the  Software  Engineering  Institute  (SEI)  of  the  Carnegie -Melon 
University,  for  software  development  activities.  Also,  this  section  will  highhght  the  importance  of 
this  taxonomy  related  to  this  study. 

As  seen  in  the  last  section,  the  implementation  of  SIEOMS  in  the  MoD  can  fit  the 
definition  of  a  project  and  the  decision  of  whether  or  not  to  proceed  with  the  implementation 
may  rely  upon  the  risk  assessment  and  identification  of  such  activity,  hr  most  organizations  and 
business  administration  when  decision  makers  are  deciding  about  an  investment  or 
implementation  of  a  new  service  or  activity  they  have  to  make  sure  that  such  initiative  will  have  a 
reasonable  chance  of  success.  Beside  other  considerations,  as  resources  available,  and  firm’s 
strategy,  they  have  to  assess  the  risks  involved  in  the  activity  in  order  to  decide  whether  is 
worthy  to  start  the  activity  or  whether  is  better  to  consider  another  alternatives  to  the 
particular/identified  need.  As  Perry  states  in  “Effective  Methods  for  Software  Testing”: 
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Risk  is  the  probability  that  undesirable  events  will  occur.  These  undesirable 
events  will  prevent  the  organization  from  successfully  implementing  its  business 
initiatives.  For  example,  there  is  the  risk  that  the  information  used  in  making 
business  decisions  will  be  incorrect  or  late.  If  the  risk  turns  into  reahty  and  the 
information  is  late  or  incorrect,  an  erroneous  business  decision  may  cause  a  failed 
business  initiative.  (1 1 :7) 

According  to  the  risks  involved  the  activity  may  or  may  not  be  implemented  and  the 

result  of  such  decision  may  be  cmcial  for  the  organization’s  future  performance.  Then,  it  is 

important  to  know  how  to  identify  risks  according  to  a  methodology  that  assures  that  key 

factors  are  being  considered  in  the  risk  assessment.  The  risk  identification  helps  to  better 

understand  what  can  jeopardize  the  project  by  allowing  the  adoption  of  measures  that  can 

attenuate  its  effects  or  simple  by  avoiding  the  risks. 

The  Software  Engineering  Institute  (SEI)  of  Carnegie-  Mellon  University  has  developed 

a  risk  identification  method  used  to  assess  risks  in  software  development.  The  SEI  taxonomy  of 

software  development  maps  the  characteristics  of  this  type  of  activity  and  the  consequent 

software  development  risks.  In  the  particular  situation  of  software  development  project, 

according  Carr  J.,  in  Taxonomy- Base  Risk  Identification  (1)  the  risks: 

(...)  can  be  known,  unknown,  or  unknowable.  Known  risks  are  those  that  one 
or  more  project  personnel  are  aware  or  -  if  not  exphcitly  as  risks.  At  least  as 
concerns.  The  unknown  risks  are  those  that  would  be  surfaced  (i.e.,  become 
known)  if  project  personnel  were  given  the  right  opportunity,  cues,  and 
information.  The  unknowable  risks  are  those  that,  even  in  principle,  none  could 
foresee.  Hence  these  risks,  while  potentially  critical  to  project  success,  are 
beyond  the  purview  of  any  risk  identification  method.  (1:7) 
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This  concepts  and  the  use  of  such  taxonomy  relates  to  this  study  in  the  sense  that  the 


use  of  this  methodology  can  be  useful  in  the  assessment  of  the  risks  involved  in  the 
implementation  of  SILOMS  in  the  MoD. 

This  subsection  has  presented  the  Taxonomy- Based  Risk  Identification  Method, 
proposed  by  a  report  of  the  Software  Engineering  Institute  (SEI)  of  the  Carnegie -Melon 
University,  for  software  development  activities  and  highhghted  the  importance  of  this  taxonomy 
related  to  this  study. 

Taxonomy  Based  Questionnaire  for  Software  Development 
This  subsection  will  present  the  Taxonomy- Based  Questionnaire  derived  from  the 
Taxonomy- Risk  Identification  Method  presented  in  the  last  section.  Also,  this  section  will 
hi^ili^t  the  importance  of  semi-  stmctured  interviews  while  yielding  a  more  vahd  data  in  risk 
assessment  for  software  development. 

The  Taxonomy- Based  Identification  Risks  is  a  repeatable  method  for  identifying  risk  in 
software  projects  using  a  software  risk  taxonomy  and  associated  questionnaire.  It  uses 
basically  a  Taxonomy- Based  Questionnaire  (TBQ),  which  consists  of  a  fist  of  non-judgmental 
questions  to  ehcit  issues  and  concerns  and  the  related  risks  in  each  taxonomic  group  - 
Appendix  C  has  a  example  of  a  TBQ.  The  use  of  the  questionnaire  guarantees  that  aU  identified 
risks  are  taken  in  account: 

(...)  the  questionnaire  ensures  that  all  risk  areas  are  systematically  addressed, 
while  the  apphcation  process  is  designed  to  ensure  that  the  questions  are  asked 
of  the  right  people  and  in  the  right  manner  to  produce  optimum  results  .  (1:7) 
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The  TBQ  application  is  semi-stmctured  and  the  questions  are  used  as  a  defining  but  not 


as  a  limiting  instrument  in  the  way  that  allows  the  discussion  to  be  made  without  concerns  with 

the  already  given  sequence  of  questions.  The  non- restriction  of  the  sequence  permits  the 

assessment  of  more  subjective  issues  as  in  a  structured  brainstorming  process.  Yet,  it  yields 

more  valid  data  according  Suchman  in  “Interactional  Troubles  in  Face- to- Face  Survey 

Interviews”  reported  by  Marvin  J.  Carr  in  Taxonomy- Based  Risk  Identification: 

This  is  done  (no-restriction  of  sequence)  to  permit  context- and- culture- sensitive 
issues  to  arise  in  as  “natural”  a  manner  as  possible.  A  completely  structured 
interview,  while  arguably  yielding  more  rehable  data  for  subsequent  analysis 
across  different  projects,  may  also  yield  less  valid  data.  (1:8) 

In  order  to  provide  a  framework  to  the  apphcation  of  the  risk  identification  method  is 

fundamental  the  understanding  of  the  software  development  taxonomy  developed  by  SEI.  The 

software  development  taxonomy: 

(...)  serves  as  the  basis  for  ehciting  and  organizing  the  full  breadth  of  software 
development  risks  -  both  technical  and  non- technical.  The  taxonomy  also 
provides  a  consistent  framework  for  the  development  of  other  risk  management 
methods  and  activities.  (1:08) 

The  software  taxonomy  is  organized  into  three  major  classes  (a  three  level  taxonomy), 
that  is,  product  engineering,  development  environment  and  program  constraints.  The  three 
major  classes  then  are  divided  into  elements,  which  in  turn  are  characterized  by  their  attributes. 
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Table  2.  Complete  3-level  SEI  Software  Development  Risk  Taxonomy 


A.  Product  Engineering 

B.  Development  Environment 

C.  Program  Constraints 

1. 

Requirements 

1. 

Development  Process 

1. 

Resources 

a. 

Stability 

a. 

Formality 

a. 

Schedule 

b. 

Completeness 

b. 

Suitability 

b. 

Staff 

c. 

Clarity 

c. 

Process  Control 

c. 

Budget 

d. 

Validity 

d. 

Familiarity 

d. 

Facilities 

e. 

Feasibility 

e. 

Product  Control 

2. 

Contract 

f. 

Precedent 

2. 

Development  System 

a. 

Type  of  Contract 

& 

Scale 

a. 

Capacity 

b. 

Restrictions 

2. 

Design 

b. 

Suitability 

c. 

Dependencies 

a. 

Functionality 

c. 

Usability 

3. 

Program  Interfaces 

b. 

Difficulty 

d. 

Familiarity 

a. 

Customer 

c. 

Interfaces 

e. 

Reliability 

b. 

Associate 

d. 

Performance 

f. 

System  Support 

Contractors 

e. 

Testability 

e. 

Deliverability 

c. 

Subcontractors 

f. 

Hardware  Constraints 

3. 

Management  Process 

d. 

Prime  Contractor 

& 

Non-Developmental 

a. 

Planning 

e. 

Corporate 

Software 

b. 

Project  Organization 

Management 

3. 

Code  and  Unit  Test 

c. 

Management 

f. 

Vendors 

a. 

Feasibility 

Experience 

g- 

Politics 

b. 

Testing 

d. 

Program  Interfaces 

c. 

Coding/Implementation 

4. 

Management  Methods 

4. 

Integration  and  Test 

a. 

Monitoring 

a. 

Environment 

b. 

Personnel 

b. 

Product 

Management 

c. 

System 

c. 

Quality  Assurance 

5. 

Engineering  Specialties 

d. 

Configuration 

a. 

Maintainability 

Management 

b. 

Reliability 

5. 

Work  Environment 

c. 

Safety 

a. 

Quality  Attitude 

d. 

Security 

b. 

Cooperation 

e. 

Human  Factors 

c. 

Communication 

f. 

Specifications 

d. 

Morale 

Carr,  as  follow,  describes  the  definition  of  each  class. 

1 .  Product  Engineering.  The  technical  aspects  of  the  work  to  be  accomphshed. 

2.  Development  Environment.  The  methods,  procedures,  and  tools  used  to 
produce  the  product. 

3.  Program  Constraints.  The  contractual,  organizational,  and  operational  factors 
within  which  the  software  is  developed  but  which  are  generally  outside  of  the 
direct  control  of  the  local  management.  (1:8) 
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The  complete  SEI  Software  Development  Risk  Taxonomy  is  presented  in  Table  2  and  a 


summary  of  Software  Development  Risk  Taxonomy  is  represented  in  Figure  5. 


This  subsection  has  presented  the  Taxonomy- Based  Risk  Identification  Method,  its 
derived  Taxonomy-Based  Questionnaire  for  software  development  activities  and  the  relation 
with  this  study.  Also  it  has  highhghted  the  importance  of  semi-stmctured  interviews  while 
yielding  a  more  valid  data  in  risk  assessment  for  software  development. 

Project  Force  Field  Analysis 

This  subsection  describes  the  method  named  as  “Force  Field  Analysis”  proposed  by 
Kurt  Fewin  in  Field  Theory  Analysis  (8). 

Fewin  (8)  has  proposed  a  method  for  analyzing  problem  situations  and  determining 
alternative  courses  of  action  by  organizing  information  pertaining  to  organizational  improvements 
into  two  categories:  those  ‘Torces”  at  work  that  restrain  improvement,  and  those  that  facilitate 
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it.  In  the  theory,  he  states  that  the  state  of  affairs  of  any  situation  is  allowed  to  persist  due  to  the 


faet  that  restraining  and  faeihtating  forees  are  in  equihbiium.  In  the  ease  of  restraining  of  forees 
oeeur  to  inerease,  then  the  state  of  affairs  wiU  worsen.  On  the  other  hand,  in  the  ease  of 
faeihtating  forees  are  strengthened  the  state  of  affairs  wiU  improve  -  Figure  6. 


Force  Field  Analysis 
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Figure  6.  Foree  Field  Analysis.  Extraeted  from  (10:548) 


The  “Foree  Field  Analysis”  uses  a  diehotomy  of  forees  to  determine  the  best  way  to 
improve  a  given  situation  by  identifying  ah  of  the  restraining  and  faeihtating  forees  and  the  relative 
strength  of  eaeh.  Then,  aeeording  the  theory  is  possible  to  determine  whieh  restraining  forees 
must  be  weakened  or  whieh  faeihtating  forees  must  be  strengthened  to  move  the  situation 
toward  the  ideal  state.  The  teehnique  was  oiiginahy  proposed  as  a  means  for  overeoming 
resistanee  to  ehange,  but  as  states  Nieholas: 
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(. . .)  it  can  be  used  by  managers  in  other  applications.  In  project  management, 
the  technique  can  be  used  to  investigate  forces  acting  on  a  current  project  or  that 
might  influence  an  upcoming  project,  and  to  determine  where  emphasis  is  needed 
to  increase  the  project’s  hkehhood  for  success.  (10:548-549) 


Some  important  considerations  have  to  be  made  by  observing  Figure  6.  First  of  aU,  the 

forces  acting  in  the  system  are  potentially  either  facihtating  or  restraining,  meaning  that  an 

specific  factor  may  be  considered  a  restraining  force  when  it  is  lacking  but  may  be  considered 

facilitating  force  when  it  is  present.  Also,  when  there  is  a  weak  or  no  presence  of  a  force 

related  to  that  factor,  then,  it  becomes  a  restraining  force  leading  to  the  “worst  state”. 

However,  if  the  force  related  to  that  factor  is  present,  its  facihtating  influence  depends  on  its 

strength  and  visibihty.  Second,  not  ah  forces  related  to  factors  are  equal;  some  are  of  generally 

greater  importance  and  influence  than  others.  FinaUy,  the  forces  are  not  always  independent  in 

the  sense  that  improving  or  strengthening  some  facihtating  forces  may  have  a  ripple  effect  on 

other  facihtating  forces.  The  implementation  of  the  analysis  is  described  in  Nicholas  as  fohows: 

A  force  field  analysis  can  be  used  in  particular  cases  for  determining  which  forces 
might  hinder  a  new  project,  or  for  analyzing  the  forces  acting  on  a  current 
project.  The  value  of  the  technique,  even  if  not  strictly  fohowed,  is  that  it 
systematizes  thinking  and  organizes  information  about  project  problems  and 
causes.  The  analysis  begins  by  gathering  information  through  questionnaires  or 
interviews  about  the  forces  facihtating  and  hindering  the  project  performance. 

(...)  The  forces  then  are  ranked  so  that  the  strongest  are  given  highest  priority. 
(10:550) 

The  implementation  involves  other  steps,  like  rating  the  forces  according  their 
“solvabhity”,  and  then  generating  actions  for  reducing  the  “solvable”  restraining  forces  with  the 
highest  priority,  which  was  given  in  the  previous  steps.  Nicholas  states  that: 
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The  utility  of  the  foree  field  analysis  proeess  is  the  systematic  framework  it 
provides  for  viewing  problems  and  identifying  solutions  with  the  highest  likelihood 
of  success.  (10:550) 

In  the  implementation  of  SILOMS  in  the  MoD,  the  identification  of  facilitating  and 
restraining  forces  can  help  to  built  a  framework  to  provide  the  solutions  of  even  to  make 
possible  the  assessment  of  the  risks  involved  in  such  activity. 

This  subsection  has  presented  the  method  Force  Field  Analysis  as  a  possible  technique 
to  be  used  in  the  assessment  of  risks  in  SILOMS’s  implementation  in  the  MoD. 


Software  Engineering  Risk  Model  -SERIM 

This  subsection  describes  the  Software  Engineering  Risk  Model  (SERIM)  described  by 
Karolak  in  Software  Engineering  Risk  Management  (6).  This  model  will  be  described  as  a  way 
to  implement  the  Taxonomy- Based  Risk  Identification  Method  (TRI)  -  a  software  risk 
identification  method. 

Choices  exist  when  making  decisions  concerning  risks  on  software  projects.  These 
choices  have  to  be  evaluated  in  a  way  to  help  the  decision  makers  to  assess  alternatives 
available  in  a  given  scenario.  According  Karolak(6: 121): 

•  The  firsts  step  is  to  analyze  alternatives  -  Alternatives  must  exist  when 
deciding  activities  based  on  risks. 

•  The  second  step  is  to  create  a  model  which  will  evaluate  alternatives.  The 
model  should  help  in  the  decision  making  process  by  assessing  the 
alternatives. 

•  The  third  step  is  to  make  a  choice.  If  a  choice  is  not  made,  the  passing  of 
time  will  dictate  the  choices  for  you. 
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The  Software  Engineering  Risk  Model  is  based  on  a  premise  that  software  development 


management  alternatives  are  always  present.  SERIM  uses  the  form  of  a  probabihty  tree 
addressing  decisions  alternatives  and  the  use  of  probabihties.  The  model  uses  the  mathematics 
of  probabihty  and  uses  it  concepts  to  address  the  likelihood  that  an  occurrence  of  event  A  lies 
within  the  sample  space  S,  where  S  is  the  hst  of  ah  possible  outcomes  of  events.  Normal  mles 
of  probabihty  hold  (6:121- 122): 

a)  P(A)  is  the  probabihty  of  event  A, 

b)  0<P(A)<  1, 

c)  P(S)  =  1,P(0)  =  0, 

d)  If  Al,  A2, ...  An  is  a  sequence  of  mutuahy  exclusive  events,  then 
P(A1  uA2u. .  .uAn)  =  P(A1)  -i-  P(A2)  -i-. .  .P(An) 

SERIM  uses  a  subjective  Bayesian  probabihty  approach  to  assess  software  risks.  This 

approach  assigns  a  subjective  probabihty  based  on  previous  experience  or  analogy  to  past 

events,  that  is,  a  personal  view  measuring  the  likelihood  or  reasonableness  that  event  A  wih 

occur.  It  is  interesting  to  note  that  if  more  than  one  person  assesses  the  subjective  probabihty, 

then,  different  results  may  be  expected.  As  stated  by  Karolak: 

Eor  two  events,  A  and  B,  P(A)  is  greater  than  or  equal  to  P(B)  if  and  only  if  A 
was  considered  to  be  more  likely  than  B.  In  this  approach,  probabihty  is  a 
measure  of  the  belief  one  has  in  the  occurrence  of  and  event.  Eor  SERIM,  the 
assignment  of  numeric  values  to  software  risk  metric  questions  shared  this  same 
subjectivity  in  the  sense  that  different  persons  may  end  up  with  different  values 
based  on  their  past  and  diverse  experiences,  business  products,  and  software 
development  environments  for  what  they  are  assessing.  As  such,  the  probabihty 
assigned  to  an  event  need  not  to  be  a  constant  value  but  can  change  based  on 
additional  experience.  (6:121) 
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The  numeric  values  used  in  SERIM  are  set  by  the  responses  to  the  metric  questions 


(according  to  the  taxonomy  of  risk  identification  adopted)  defined  to  perform  the  interviews. 
Based  on  the  responses  to  the  questions  having  a  value  between  0  and  1,  the  probabihty  of  risks 
can  be  computed.  Then,  probabihty  trees  are  used  to  calculate  an  overall  success  rate,  which  is 
a  weighted  average  of  the  probability  of  events  associated  with  the  risk. 

SERIM  relates  risk  metrics  to  software  life  cycle  phases  and  software  risk  management 
achvities.  By  doing  so,  software  risk  can  be  identified  by  the  phase  of  the  software 
development  and  correlated  to  each  of  the  metric  questions  used  in  the  risk  identification 
method. 

Eikewise,  the  probabihty  for  each  hfe  cycle  phase,  risk  factors,  risk  elements, 
and  risk  management  activities  can  be  represented  as  a  probabihty  tree  based  on 
the  answer  to  the  metric  questions.  (6: 123) 

Total  Product  Risk  P(A) 

Software  Project  Risk 

Technical  Cost  Schedule 

Risk  Elements  P(A1)  P(A2)  P(A3) 


Risk  Factors  Organization  Estimation  Monitoring  .  Personnel 

P(A4)  P(A5)  P(A6)  P(A13) 


Risk  Metrics  01  02  03  05  06  07  08  09  01 0  011  012  .  On 

Eigure  7.  Partial  Software  Risks  Relationships  from  Karolak  (6:124) 

The  example  used  in  Karolak  (6: 121- 131),  and  partially  reproduced  in  Eigure  7,  shows 
the  relationships  within  risk’s  parameters  and  Table  3  and  Table  4  shows  the  detailed  risks’ 
parameters,  where: 
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•  P(A)  represents  the  probability  of  a  successful  software  project, 

•  P(A1),  P(A2),  and  P(A3)  identify  the  likelihood  of  successMly  meeting 
future  technical,  cost,  and  schedule  goals. 

•  P(A4)  through  P(A14)  represent  the  likelihood  of  successfully  meeting  the 
software  risk  factors  identified  according  a  given  methodology  or  risk 
identification. 

•  P(B)  through  P(G)  represent  the  likelihood  of  a  successful  software  project 
based  on  the  phase  the  software  development  life  cycle  of  the  project. 

•  P(H)  through  P(M)  identify  the  probabihty  of  meeting  the  software  risk 
management  activities  previously  identified. 

To  implement  SERIM,  several  parameters  and  equations  must  be  identified  and 

considered.  The  following  equations  are  used  for  each  of  the  probabihty  trees  according  the 

example  given  by  Karolak  (6:121-131): 

1)  P(A)  =  [E\=i  P(An)]/3  assuming  that  each  risk  element  is  equal  in  weight.  If 
the  weight  of  each  element  differs  between  them,  then  P(A)  =  wiP(Ai)  = 
W2P(A2)  +  WsPCAs)  where  each  Wi  is  a  positive  number  and  Wi  -i-  W2  -i-  W3  =  1. 

2)  P(Element)  =  [E'^n=4  WnP(An)]  where: 

a.  An  is  the  metric  value  for  the  factors  identified  in  Table  3,  and  related  to 
the  element  being  measured 

b.  Wn  is  the  weight  assigned  according  risk  factor’s  influence  against  risk 
elements. 

3)  P(Eactor)  =  [E‘’n=i  P(Qn)]/8  where  Qn  is  the  metric  value  for  the  question 
number  Qn  identified  as  related  to  the  factor  being  measured. 

4)  P(Development  Phase)  =  Z(AU  values  assigned  to  the  questions  related  to  the 
developmental  phase)/number  of  questions. 

5)  P(Software  Management  Activity)  =  Z(Atl  values  assigned  to  the  questions 
related  to  the  software  management  activity )/number  of  questions. 
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Table  3.  Sw  Risks  According  the  Example  Extracted  from  (6:121- 131) 


Software  Risk  Elements 

At 

Technical 

A2 

Cost 

A3 

Schedule 

Software  Risk  Factors 

A4 

Organization 

A5 

Estimation 

A6 

Monitoring 

A7 

Development  Methodology 

A8 

Tools 

A9 

Risk  Culture 

AlO 

Usability 

All 

Correctness 

A12 

Reliability 

A13 

Personnel 

Table  4.  Sw  Development  Phases  and  Risk  Management  Activities  -  (18) 


Project’s  Software  Developmental  Life  Cycle  -  Phases 

B 

Pre-Requirement 

C 

Requirements 

D 

Design 

E 

Code 

F 

Test 

G 

Development  and  Maintenance 

Software  Risk  Management  Activities 

H 

Identification 

I 

Strategy  and  Planning 

J 

Assessment 

K 

Mitigation/ Avoidance 

L 

Reporting 

M 

Prediction 

By  adding  SERIM  to  TRI,  both  turns  into  a  software  risk  identification  method 
supported  by  a  tool  that  makes  possible  the  consohdation  of  the  software  risk  information 
gathered  by  the  use  of  the  TBQ.  SERIM  will  tie  the  relationships  between  all  the  software  risk 
information  available  in  order  to  help  the  decision  makers  in  addressing  decisions  alternatives 
through  probabihties. 
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This  subsection  has  described  SERIM  as  a  way  to  implement  the  Taxonomy- Based 


Risk  Identification  Method  (TRI)  -  a  software  risk  identification  method. 

Chapter  Summary 

This  chapter  has  presented  the  definitions  and  characteristics  of  a  qualitative  research  as 
well  as  considerations  about  surveys  and  the  use  of  questionnaires,  interviews  and  Likert’s  scale 
and  its  application  to  the  present  study.  Second,  a  literature  review  of  the  relationship  between 
projects  and  risks,  as  well  as  the  appropriateness  of  using  project  management  approach  in 
activities  with  inherent  risks  associated  were  discussed.  Third,  the  chapter  discussed  and 
presented  the  concepts  of  managing  risks  in  software  development  project  and  available  risk 
analysis  methods.  Fourth,  the  ‘Taxonomy-Based  Risk  Identification”,  a  method  to  risk 
identification  in  software  development  environments,  and  the  derived  Taxonomy-Based 
Questionnaire  (TBQ)  were  presented.  Fifth,  Force  Field  Analysis  was  presented  as  a 
technique  available  for  analyzing  problem  situations.  Finally  the  SERIM  model  was  presented 
as  a  way  to  implement  the  TRI. 
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III.  Methodology 


Introduction 

This  chapter  describes  the  procedures  taken  during  the  research  process  to  achieve  its 
objectives.  Describing  the  research  design,  and  data  analysis  method,  the  chapter  wUl  end  up  in 
proposing  a  method  to  answer  the  Investigative  Questions  stated  in  Chapter  1  and 
consequently,  also  end  up  by  providing  means  to  answer  the  Research  Question. 

In  order  to  accomphsh  this  goal,  first,  this  chapter  wiU  present  the  research  design  and 
data  analysis  method,  that  is,  a  combination  of  a  quahtative  and  quantitative  methods  using  a  mix 
and  adapted  tools  described  in  Chapter  n,  to  gather  and  analyze  the  data  obtained  in  the 
research  process.  The  first  section  is  subdivided  into  two  subsections  describing  the 
methodology  used  in  each  quahtative  and  quantitative  portions  of  this  study.  Second,  this 
chapter  wiU  present  the  population  involved  in  the  study  as  weU  as  the  samphng  information, 
which  consists  of  a  few  carefuUy  selected  agencies  in  the  MoD.  Third,  this  chapter  describes 
the  nature  of  the  data  involved  in  the  study.  Finahy  a  brief  summary  wiU  be  presented. 

Research  Design  and  Data  Analysis 

This  section  describes  the  research  design  chosen  to  perform  this  study,  a  combination 
of  quahtative  and  quantitative  analysis  of  the  organizational,  cultural  and  pohtical  aspects  that  can 
threat  the  successful  implementation  of  SILOMS  in  the  MoD’s  agency  SELOM.  It  also 
describes  the  data  analysis  method  used  to  assess  the  risk  associated  with  this  implementation. 
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As  shown  in  the  literature  review  in  Chapter  n,  quahtative  methods  have  the  advantage 


of  going  in  depth  in  the  problem  by  allowing  more  flexibihty  to  the  researcher  in  a  real  world 
environment  or  natural  setting  with  inherent  subjective  aspects.  Also,  qualitative  research  is 
more  adequate  to  explore  human  events  that  have  inherent  subjectivism  and  where  multiple 
perspectives  can  be  held  by  different  individuals.  These  multiple  perspective,  in  turn,  may  have 
equal  validity  or  add  value  to  the  data  analysis  and  conclusions.  Furthermore,  these  multiple 
perspectives  and  reveahng  their  natures  end  up  of  being  one  important  goal  of  quahtative 
studies. 


The  quahtative  porhon  of  this  study  wih  be  performed  using  a  combination  of  two  of  the 
five  common  designs  to  qualitative  studies,  described  in  Chapter  n,  that  is.  Case  Study, 
Ethnography,  Phenomenological  Study,  Grounded  Theory  Study  or  Content  Analysis.  The 
chosen  design  is  a  combination  of  Case  Study  and  Phenomenological  Study.  The  first  one  is 
described  in  Leedy  as: 

In  a  case  study,  a  particular  individual,  program,  or  event  is  studied  in  depth  for  a 
defined  period  of  time.  (...)  A  case  study  may  be  especiahy  suitable  for  learning 
more  about  a  tittle  known  or  poorly  understood  situation.  It  may  also  be  useful 
for  investigating  how  an  individual  or  program  changes  over  time,  perhaps  as  the 
result  of  certain  circumstances  or  interventions.  (...)  The  researcher  also  records 
details  about  the  context  in  which  the  case  is  found,  including  information  about 
the  physical  environment  and  any  historical,  economic,  and  social  factors  that 
have  bearing  on  the  situation.  (7 : 149) 

The  second  one  is  described  in  Leedy  as: 

In  its  broadest  sense,  the  term  phenomenology  refers  to  a  person’s  perception  of 
the  meaning  of  an  event,  as  opposed  to  the  event  as  it  exists  external  to  the 
person.  A  Phenomenological  study  is  a  study  that  attempts  to  understand 
people’s  perceptions,  perspectives,  and  understandings  of  a  particular  situation. 

(7:153) 
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The  combination  of  Case  Study  and  Phenomenological  Study  were  chosen  because 


these  designs  seems  to  be  complement  each  other  and  this  mix  is  more  appropriate  to  fit  the 

research  objective  of  understanding  the  risk  associated  to  organizational,  cultural  and  pohtical 

aspects  that  can  threat  the  successful  implementation  of  SILOMS  in  the  MoD’s  agency 

SELOM.  Furthermore,  it  is  important  to  gather  people’s  perceptions  about  the  implementation 

of  SILOMS  and  look  for  hints  and  issues  that  can  be  viewed  as  a  threat  to  a  successful 

implementation.  This  kind  of  design  has  interesting  characteristics  as  pointed  out  by  Leedy: 

The  actual  implementation  of  a  phenomenological  study  is  as  much  in  the  hands 
of  the  participants  as  in  the  hands  of  the  researcher.  The  phenomenological 
interview  is  often  a  very  unstmctured  one  in  which  the  researcher  and  the 
participants  work  together  to  “arrive  at  the  heart  of  the  matter”  (Tesch,  1994,  p. 

147).  The  researcher  listens  closely  as  participant  describe  their  everyday 
experiences  related  to  the  phenomenon  and  must  be  alert  for  subtle  yet 
meaningful  cues  in  participants’  expressions,  questions,  and  occasional 
sidetracks.  A  typical  interview  looks  more  tike  an  informal  conversation,  with 
the  participant  doing  most  of  the  talking  and  the  researcher  doing  most  of  the 
listening.  (7:153) 

On  the  other  hand,  a  qualitative  assessment  only  may  be  not  sufficient  to  give  an 
objective  evaluation  of  the  risks  involved  in  the  implementation  of  SILOMS  in  the  MoD.  A 
quantitative  analysis  applies  to  this  case  in  the  way  that  it  is  necessary  to  quantify  the  risks 
involved  in  the  implementation  of  SILOMS  in  the  MoD  by  assigning  probabihties  for  the 
identified  risks.  The  assignment  of  probabihties  may  come  from  different  basis,  like 
experimental  evidence,  expert  opinion,  subjective  judgment.  In  any  case  the  value  added  to 
data  analysis  is  worthy.  Furthermore,  when  probabihties  are  assigned,  some  sort  of  a 
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quantitative  analysis  is  needed  either  to  allow  further  comparisons  among  available  alternatives 


or  either  by  simple  measuring  the  probabihty  of  success  or  failure  of  a  unique  situation. 

This  study  has  rehed  upon  an  adaptation  of  the  Taxonomy-Based  Identification  Risk 
Method  (TRI),  and  derived  Taxonomy-Based  Questionnaire  (TBQ)  proposed  by  Carr  in 
Taxonomy-Based  Risk  Identification  (I),  to  undertake  interviews  with  key  personnel  in 
SELOM,  the  MoD’s  agency.  The  questionnaire  used  in  the  interviews  was  designed  mainly  to 
gather  qualitative  and  quantitative  data  and  achieve  two  goals: 

•  Gather  data  concerning  peoples’  perceptions  about  issues  related  to  risks  in 
software  development  projects,  through  the  use  of  open-ended  questions. 

•  Gather  data  related  to  the  risks  factors,  attributes  and  elements,  under  the 
adapted  taxonomy  or  risk  identification,  through  the  use  of  objective  or 
standard  questions. 

The  methods  used  to  gather  and  analyze  both  quahtative  and  quantitative  data  will  be 
described  separately  in  the  next  subsections.  First,  the  methodology  used  to  gather  and  analyze 
the  qualitative  portion  will  be  presented.  Second,  there  will  be  a  presentation  about  the  way  the 
quahtative  assessment  turns  into  a  quantitative  measure  to  provide  an  objective  assessment  of 
the  feasibihty  of  SILOMS’s  implementation  in  the  MoD. 

Qualitative  Design  Portion  Methodology 

In  order  to  perform  the  qualitative  portion  of  this  research,  this  study  has  rehed  upon  an 
a  adaptation  of  the  Taxonomy- Based  Risk  Identification  Method  (TRI)  and  its  derived 
Taxonomy-Base  Questionnaire  (TBQ)  proposed  by  Carr  in  Taxonomy- Based  Risk 
Identificationd).  The  TRI  method  and  the  TBQ  had  been  used  as  a  basis  to  the  development 
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of  the  MoD-SILOMS  Taxonomy  Risk  Identification  Method  (MSTRI)  -  Figure  8  -  and  its 
derived  SILOMS  Taxonomy-Based  Questionnaire  (MSTBQ)  -  see  Appendix  A. 


Figure  8.  MSTRI  -  Taxonomy  Adapted  from  Carr  (I) 


The  MSTRI  is  a  taxonomy  adapted  to  fit  the  special  case  considered  in  this  study,  that 
is,  the  implementation  of  SILOMS  in  the  MoD.  The  derived  MSTBQ  is  a  semi-stmctured 
interview  based  on  a  mix  of  open-ended  and  objective  or  standard  questions.  The  researcher 
has  performed  the  interviews  with  key  personnel  in  pre-selected  MoD’s  agencies  -  see  Figure 
1 1 .  The  interviews  had  the  objective  of  gathering  data  concerning  parameters,  such  as  the 
proposed  risk  factors,  attributes  and  elements,  according  to  MSTRI.  These  parameters  were 
selected  according  to  the  researcher’s  experience  on  the  field,  observations  and  also  in  expert 
opinions  found  in  the  hterature,  by  being  common  problems/issues,  which  may  have  potential 
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effects  over  a  software  development  project  such  as  the  implementation  of  SILOMS  in  the 
MoD. 


Questionnaire 

The  MSTBQ  used  to  perform  the  interviews  has  five  sections;  the  first  section  consists 
of  an  explanation  to  the  interviewees  about  the  SILOMS  project.  The  second  section  consists 
of  an  explanation  of  the  context  and  purpose  of  the  research.  The  third  section’s  questions 
relate  to  demographic  data.  The  fourth  section  explains  the  scoring  method  for  the  objective  or 
standard  questions  consisting  of  the  use  of  a  rating  scale  and  a  rank  order  procedure  which  are 
basically  the  use  of  Likert  scales  and  the  assignment  of  weights  -  given  by  interviewees  - 
according  to  the  relative  importance  of  the  parameter  compared  to  others.  The  fifth  and  last 
section  contains  the  definitions  of  the  Program  Constraints  Class  defined  under  the  MSTRI 
and  the  questions  within  MSTBQ,  which  is  the  main  source  of  data  in  this  research,  as  will  be 
described  in  further  section  in  this  Chapter. 

The  open-ended  questions  were  used  to  assess  the  interviewee’s  subjective  opinions 
and  perceptions  to  specific  issues  related  to  software  development  knowledge  and  about 
SILOMS.  This  kind  of  questions  helped  the  researcher  to  gather  information  and  draw 
conclusions  that  would  not  be  possible  in  either  objective  or  standard  questions. 

In  addition,  objective  or  standard  questions  were  used  to  gather  more  specific  opinions 
in  a  way  that  they  could  be  assigned  numerical  measures  values,  according  to  the  rating  scale,  to 
further  help  the  data  analysis.  These  objective  or  standard  questions  were  the  parameters  in 
which  the  quantitative  portion  of  the  research  was  performed. 
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Since  the  MSTRI  is  a  taxonomy  that  depicts  factors,  attributes  and  elements  that  can 


turn  into  risks  or  constraints  to  the  implementation  of  SILOMS  in  the  MoD,  then  in  this  point, 
we  can  say  that  the  MSTRI  and  its  derived  MSTBQ  is  a  tool  used  in  the  research  to  answer  the 
Investigative  Question: 

•  What  are  the  factors  critical  to  the  successful  implementation  of  SILOMS  in  the 
MoD? 

The  remaining  investigative  questions  will  be  answered  by  the  use  of  a  methodology 
described  in  the  following  section. 

Quantitative  Design  Portion  Methodology 

This  study  rehed  upon  the  MSTBQ  to  gather  quantitative  data  through  the  use  of 
objective  and  standard  questions  related  to  the  risks  factors,  attributes,  and  elements  under  the 
Program  Constraints  Class,  defined  in  the  MSTRI  and  shown  in  Figure  10. 

A  quantitative  analysis  apphes  to  this  case  in  the  way  that  it  is  necessary  to  quantify  the 
risks  involved  in  the  implementation  of  SILOMS  in  the  MoD  by  assigning  probabihties  for  the 
identified  risks.  The  assignment  of  probabihties  may  come  from  different  basis,  like 
experimental  evidence,  expert  opinion,  subjective  judgment.  Once  probabihties  were  assigned, 
quantitative  analysis  is  needed  either  to  ahow  further  comparisons  among  available  alternatives 
or  by  simple  measuring  the  probabihty  of  success  or  failure  of  a  unique  situation. 

In  the  particular  case  of  this  study,  the  probabihties  were  assigned  to  the  objective  or 
standard  questions  -  described  in  the  last  section  as  parameters  -  within  the  MSTBQ.  These 
represented  risk  factors,  attributes  and  elements  related  to  the  MSTRI.  Each  attribute  was 
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translated  into  questions  in  the  interview  form  of  the  MSTBQ  -  see  Appendix  A.  Then  a  rating 


scale  was  used  to  assign  numbers,  within  a  range  of  values,  based  on  a  general  scale  shown  in 
Figure  9. 


(  ) 

{  ) 

( ) 

{  ) 

0 

0.2 

0.8 

1.0 

None 

A  Little 

Most 

All 

Figure  9.  Scale  as  a  general  reference 


These  values  of  probabihties  were  assigned  using  a  subjective  judgment  of  the 

researcher  and  interviewees,  as  well  as  expert  opinions  found  in  the  hterature.  The  subjective 

judgment  was  then,  translated  into  probabihty.  As  states  Fabriky: 

Decision  making  under  risk  occurs  when  the  decision  maker  does  not  suppress 
acknowledged  ignorance  about  the  future,  but  makes  it  exphcit  through  the 
assignment  of  probabilities.  Such  probabilities  may  be  based  on  experimental 
evidence,  expert  opinion,  subjective  judgment,  or  a  combination  of  these. 

(3:102) 

The  objective  or  standard  questions  were  designed  to  measure  software  risk  factors 
that  are  designated  to  attributes,  which  in  turn  are  associated  to  elements,  and  finally  linked  to 
the  Program  Constraints  Class.  Therefore,  the  numbers  associated  to  the  factors  were  used 
as  a  way  to  quantify  risk  associated  with  each  aforementioned  parameter  within  the  MSTRI 
taxonomy  risk  identification. 

Using  concepts  of  the  Software  Engineering  Risk  Model  (SERIM)  proposed  by 
Karolak  in  Software  Engineering  Risk  Management  (6)  -  described  in  Chapter  II  -  the  numbers 
assigned  were  used  to  implement  the  risk  assessment  of  the  Program  Constraint  Class 
considering  the  hierarchy  and  relationships  within  the  risk  factors,  attribute  and  elements  under 
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the  MSTRI  -  see  Figure  10.  Then  the  eonsohdation  of  the  probabilities  assigned  to  the 


parameters  ended  up  by  being  a  number,  which  represents  the  probabihty  of  a  successful 
implementation  of  MSTRI’s  Program  Constraint  Class.  Ultimately  the  number  assigned  to 
the  Program  Constraint  Class,  within  the  defined  scope  of  this  research,  ends  up  by  being  the 
probability  of  successful  implementation  of  SILOMS  in  the  MoD. 

The  hierarchy,  relationship  and  interdependencies  within  the  risk  factors,  attributes  and 
elements  that  were  used  to  assess  the  risks  involved  in  the  Program  Constraints  Class  is 
shown  in  Figure  10. 


Figure  10.  Program  Constraint  Class  -  Parameters’  Relationship  -  MSTRI. 
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Table  5.  Detailed  List  of  Parameters’  Hierarchy 


Program  Constraints  Class 

Element  A2.4  -  Organizational  Element  Risk 

Attribute  | 

A2.4.1  -  Differences  in  Organizations’  Structures 

Factor 

A2.4.1.1 

Factor 

A2.4.1.2 

Factor 

A2.4.1.3 

Attribute  | 

A2.4.2  -  Managers  Commitment  to  Cross  Organizational  Projects 

Factor 

A2.4.2.1 

Factor 

A2.4.2.2 

Factor 

A2.4.2.3 

Attribute  | 

A2.4.3  -  Organization’s  Strategy  to  Cross-Organization  Project  Management 

Factor 

A2.4.3.1 

Factor 

A2.4.3.2 

Element  A2.5  -  Cultural  Element  Risk 

Attribute  | 

A2.5.1  -  Differences  in  Organizations’  Cultures 

Factor 

A2.5.1.1 

Factor 

A2.5.1.2 

Factor 

A2.5.1.3 

Attribute  | 

A2.5.2  -  Willingness  to  Change 

Factor 

A2.5.2.1 

Factor 

A2.5.2.2 

Element  A2.6  -  Political  Element  Risk 

Attribute  | 

A2.6.1  -  Internal  Disputes  in  Organizations’  Politics 

Factor 

A2.6.1.1 

Factor 

A2.6.1.2 

Factor 

A2.6.1.3 

Factor 

A2.6.1.4 

Factor 

A2.6.1.5 

Attribute  | 

A2.6.2  -  Feuds  Existence  in  Organizations’  Politics 

Factor 

A2.6.2.1 

Factor 

A2.6.2.2 

Another  issue  that  was  taken  into  account  in  the  interviews  when  performing  the  fifth 
section  of  the  MSTBQ  was  that  the  researcher  asked  for  the  interviewees  to  rank  order  the 
attributes,  within  each  element,  and  the  elements,  within  the  Program  Constraints  Class.  This 
was  done  to  allow  the  assignment  of  weights  to  risk  factors,  attributes  and  elements  to  reveal  the 
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importance,  according  the  interviewees’  perception,  of  the  parameters  in  the  computation  of  the 


total  risk  assessment. 

Finally,  the  SERIM  method  is,  in  conjunction  with  the  MSTRI  and  MSTBQ,  the  tool 
used  in  the  research  to  quantify  the  risks  factors,  attributes  and  elements  that  can  turn  into  a 
potential  risks  or  constraints  to  the  implementation  of  SILOMS  in  the  MoD.  Furthermore, 
these  combined  methodologies  will  be  used  in  the  research  to  answer  investigative  questions 
stated  in  Chapter  1. 

This  section  has  described  the  design  and  data  analysis  method  used  to  analyze  the  data 
gathered  from  researcher’s  observation  as  well  as  from  parameters,  within  the  MSTRI, 
assessed  in  the  interviews.  Also  it  has  described  the  way  in  which  the  qualitative  data  has 
provided  a  quantitative  assessment,  through  the  assignment  of  probabilities,  in  those  parameters 
that  represented  the  occurrence  of  a  particular  risk.  And  finally  this  section  has  presented  the 
parameters’  hierarchy,  relationship  and  interdependencies,  which  ended  up  with  a  number  that 
gives  an  objective  assessment  of  the  probabUity  of  successful  implementation  of  SILOMS  in  the 
MoD. 


Summary  of  Steps  Taken  in  the  Research  Process 

This  subsection  summarizes  aU  the  steps  taken  to  perform  this  study.  After  the 
description  of  the  design  and  data  analysis  made  in  the  previous  subsections  it  is  now  possible  to 
summarize  the  steps  taken  in  the  research  process  to  give  a  better  understanding  of  this  study. 

Also,  by  doing  so,  the  research  and  investigative  questions  can  be  related  to  these  steps 
in  the  sense  that  they  will  be  answered  along  the  performed  steps. 


50 


The  following  list  summarizes  the  steps  taken  in  the  research  design  to  answer  the 


research  and  investigative  questions: 

1 .  Using  the  Taxonomy- Based  Risk  Identification  Method  (TRI)  proposed  by  Carr  in 
Taxonomy-Based  Risk  Identification(l),  the  researcher  has  adapted  the  TRI  to  fit 
the  specific  situation  under  this  study,  that  is,  the  implementation  of  SILOMS  in  the 
MoD.  The  new  taxonomy  was  named  as  MoD- SILOMS  Taxonomy  Risk 
Identification  Method  (MSTRI)  -  see  Figure  8. 

2.  The  TRI  method  has  a  derived  Taxonomy- Based  Questionnaire  (TBQ),  and  the 
researcher  has  adapted  the  TBQ  to  fit  the  specific  situation  under  this  study.  The 
new  questionnaire  was  named  as  MoD- SILOMS  Taxonomy- Based  Questionnaire 
(MSTBQ)  -  see  Appendix  A. 

3.  The  MSTBQ  was  used  to  conduct  interviews  with  key  people  in  MoD’s  agencies 
as  highlighted  in  Figure  1 1 . 

4.  The  quahtative  and  quantitative  data  collected  in  the  interviews  were  gathered 
through  questions  designed  in  the  MSTBQ.  The  first  ones  gathered  through  the  use 
of  open-ended  questions  and  the  last  ones,  gathered  through  the  use  of  objective  or 
standard  questions. 

5.  The  open-ended  questions  were  used  to  give  the  researcher  more  flexibility  in 
gathering  data  related  to  a  few  central  issues  that  had  to  be  observed  in  the  study. 
Also,  the  open-ended  questions  gave  the  researcher  the  opportunity  of  gathering 
unexpected  information  since  the  interviewees  could  come  up  with  new  revealing 
issues  related  to  the  study. 

6.  The  objective  or  standard  questions  were  designed  and  used  to  obtain  numerical 
values,  through  the  use  of  a  rating  scale  system,  based  on  the  so-called  Likert 
scales.  Also  a  rank  order  procedure  was  performed  in  each  section  of  the  MSTBQ 
to  allow  the  assignment  of  weights,  which  were  given  by  interviewees’  opinion 
according  to  the  relative  importance  of  the  parameter  compared  to  others. 

o  The  numerical  values  were  assigned  to  the  parameters  (each  one  a  question 
itself)  being  considered  under  the  scope  of  the  research,  that  is,  the  risks 
factors,  attributes,  and  elements  under  the  Program  Constraints  Class, 
defined  in  the  MSTRI  and  shown  in  Figure  10. 
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7.  Each  parameter  aforementioned  was  strietly  related  to  the  parameter  immediately 
above  according  to  a  hierarchy  shown  in  Figure  10  and  detailed  in  Table  5.  This 
parameters’  hierarchy  of  dependencies  and  relationships,  combined  with  the 
numbers  assigned  to  them,  turned  in  a  framework  that  have  made  possible  a  overall 
numerieal  assessment  of  the  Program  Constraints  Class,  interpreted  as  a 
probability  of  suecess  of  the  implementation  of  SILOMS  in  the  MoD  -  limited  to  the 
defined  scope  of  this  study.  See  Figure  8  and  Figure  10. 

8.  The  probabihty  of  the  suecessfiil  implementation  of  the  Program  Constraints 
Class  was  then,  ealeulated  according  the  adaptation  of  SERIM’s  method  using  the 
following  formulations  -  see  Appendix  E: 

a.  P(A2)  =  [F^n=i  WnP(An)]/3  assuming  that  the  weight  of  eaeh  element  differs 
between  them,  then  P(A)  =  wiP(Ai)  =  W2P(A2)  +  W3P(A3)  where  each  Wi  is  a 
positive  number  and  Wi  +  W2  +  W3  =  1. 

b.  P(A2.n)  =  [Z^n=4  WnP(A2.n)]  where: 

i.  A2n  is  the  metric  value  for  the  factors  identified  in  Table  3,  and  related 
to  the  element  being  measured 

ii.  Wn  is  the  weight  assigned  aeeording  risk  factor’s  influence  against  risk 
elements. 

e.  P(A2.n.q)  =  [Z‘*n=i  P(Qn)]/8  where  Qn  is  the  metric  value  for  the  question 
number  Qn  identified  as  related  to  the  faetor  being  measured. 

d.  P(Development  Phase)  =  Z(A11  values  assigned  to  the  questions  related  to  the 
developmental  phase)/number  of  questions. 

e.  P(Software  Management  Activity)  =  Z(A11  values  assigned  to  the  questions 
related  to  the  software  management  activity )/number  of  questions. 

This  subsection  has  summarized  all  the  steps  taken  to  perform  this  study. 


Population  and  Sampling  Information 

This  section  describes  the  population  involved  in  this  study  as  well  as  the  sampled 
organizations  that  will  take  part  of  this  researeh. 
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The  population  will  be  the  MoD’s  logistics  agencies  and  related  Brazihan’s  Navy  (MB), 


Army  (EB),  and  Air  Force  (FAB)  organizations.  Since  the  focus  wiU  be  on  the  organization, 
cultural  and  pohtical  aspects  that  can  threat  the  successful  implementation  of  SIFOMS  in  the 
MoD’s  agency  SEFOM  and  only  explored  within  middle  and  high-level  managers  of  MoD’s 
agencies,  then,  the  sample  will  be  the  MoD’s  agencies  highhghted  in  Figure  1 1 . 


Figure  1 1 .  Brazil’s  Ministry  of  Defense  -  Highhghted  Fogistics  Agencies. 


This  section  has  described  the  population  involved  in  this  study  as  weU  as  the  sampled 
elements  and  provided  a  graphical  view  of  the  organizations  being  researched. 

Nature  of  the  Data 

This  section  describes  the  nature  of  data  gathered  in  the  research  process  as  weU  as  the 
data  coUection  method. 
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The  data  will  consist  in  qualitative  assessment  made  possible  by  researcher’s 


observation  and  the  use  of  open-ended  questions  in  the  MSTBQ  and  also  by  quantitative 
assessment  in  objective  questions  that  have  scores  associated  with. 

Data  win  be  extracted  from  researcher’s  observations,  as  a  relative  outsider  from 
interviews  with  the  managers  of  the  sectors  in  the  MoD  highhghted  in  Figure  1 1 .  The  interviews 
win  be  focused  on  the  adapted  MSTRI  and  its  derived  MSTBQ  -  Appendix  A  -  concerning  the 
Program  Constraint  Class. 

This  section  has  described  the  nature  of  the  data  gathered  in  the  research  process  as 
weU  as  the  data  collection  method,  through  MSQ,  applied  to  this  study. 

Chapter  Summary 

This  chapter  has  described  the  procedures  taken  during  the  research  process  to  achieve 
its  objectives.  It  has  described  the  research  design,  and  data  analysis  method,  in  order  to 
answer  the  Investigative  Questions  and  consequently  answering  the  Research  Question  stated 
in  Chapter  I.  Also,  this  chapter  has  presented  the  population  involved  in  the  study  as  weU  as  the 
samphng  information,  which  consisted  in  a  few  carefully  selected  agencies  in  the  MoD.  Finally, 
this  chapter  has  described  the  nature  of  the  data  involved  in  the  study. 
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IV.  Results 


Introduction 

By  giving  a  numerical  assessment  of  the  risks  factors,  attributes  and  elements  under  the 
Program  Constrain  Class  -  MSTRI  -  the  investigative  questions  will  be  answered.  Also,  the 
answers  are  expected  to  direct  the  use  of  the  method  as  a  framework  to  help  the  decision 
makers  to  decide  whether  or  not  to  implement  SILOMS  in  the  MoD’s  agency  SELOM. 

The  main  purpose  of  this  chapter  is  to  present  the  results  of  the  study  and  answer  the 
investigative  questions  described  in  Chapter  1.  The  data  obtained  through  the  use  of  MSTQ, 
from  open-ended  questions  and  objective  or  standard  question  are  presented  in  the  first  section, 
The  second  section  analyzes  and  interprets  the  data  obtained.  The  third  section  uses  the  force 
field  analysis  to  provide  an  overall  picture  of  the  forces  acting  in  the  SILOMS ’s  implementation 
in  the  MoD.  The  fourth  section  answer  the  investigative  questions  stated  in  Chapter  1.  Finally 
the  chapter  summary  is  presented. 

Results 

This  section  present  the  results  obtained  in  the  interviews  performed  with  key  personnel 
in  the  MoD’s  agencies  according  to  the  agencies  highhghted  in  Figure  1 1 .  The  first  subsection 
shows  the  data  gathered  from  the  open-ended  questions  within  the  MSTBQ.  The  second 
subsection  shows  data  gathered  from  the  objective  or  standard  questions  in  tables  that 
summarize  the  scores  obtained. 
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Open-ended  Questions  -  Additional  Issues 


This  subsection  shows  the  open-ended  questions  that  were  used  to  assess  the 
interviewees’  subjective  opinions  and  perceptions  related  to  specific  issues  of  software 
development  knowledge  about  SILOMS.  These  questions  helped  the  researcher  gather 
information  and  draw  conclusions  that  would  not  be  possible  with  other  types  of  question. 

Some  issues  were  raised  while  asking  the  interviewees  their  perception  of  SILOMS  and 
its  insertion  in  the  MoD.  This  was  done  before  asking  them  the  objective  or  standard  questions 
to  avoid  giving  the  interviewees  hints  about  the  risk  taxonomy  already  estabhshed  in  the 
MSTBQ.  The  main  points  were: 

•  Concerns  about  differences  in  cultures  within  the  military  -  raised  in  three  out  of  four 
interviews. 

•  Concerns  about  the  feasibihty  of  the  implementation  of  an  integrated  database 
integrating  the  three  mihtary  branches  and  the  rehabihty  of  such  database  -  raised  in 
one  out  of  four  interviews.  He  argued  that  even  within  a  single  mihtary  organization, 
such  a  system  would  chaUenge  the  actual  status. 

Objective  or  Standard  Questions  -  MSTBQ’s  -  Parameters  Results 

This  subsection  shows  data  gathered  from  the  objective  or  standard  questions.  Table  6 
shows  the  summary  of  the  parameter’s  ranks  given  by  the  interviewees’.  Table  7  contains  the 
scores  given  to  the  parameters  hsted  in  Table  5  -  according  the  rating  scale  and  rank  order 
processes  described  in  Chapter  III. 

Complete  tables  of  objective  questions’  scores  obtained  from  each  interview  using 
MSTBQ  are  shown  in  Appendix  D. 
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Table  6.  Summary  of  Parameters’  Ranks 


Program  Constraint  Class  -  Ranks  to  Elements 

Metric 

Element 

Inerview 

1 

2 

3 

4 

A2.1 

N/A 

N/A 

N/A 

N/A 

A2.2 

N/A 

N/A 

N/A 

N/A 

A2.3 

N/A 

N/A 

N/A 

N/A 

A2.4 

2 

3 

1 

2 

A2.5 

3 

2 

3 

3 

A2.6 

1 

1 

2 

1 
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MSTBQ  Responses 


Table  7.  MSTBQ  -  Responses  to  Objective  Questions 
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Analysis  -  Interpreting  Results 


From  Table  7  it  is  possible  to  review  the  data  and  interpret  the  meaning  of  the  scores. 
The  score  labeled  as  1-(A2),  shown  in  the  summary  table,  is  interpreted  as  the  probabihty  of 
success  related  to  the  Program  Constraints  Class,  considering  the  organizational,  cultural  and 
pohtical  aspects  of  the  implementation  of  SILOMS  in  the  MoD. 

The  interviews  were  gathered  from  four  different  subjects,  and  differences  were 
expected  and  are  due  to  the  fact  that  the  assignment  of  numeric  values  to  risk  parameters  is 
subjective  and  different  respondents  based  their  responses  upon  their  past  experiences  related 
to  software  development.  In  order  to  analyze  the  responses  it  is  necessary  to  address  each  one 
separately. 

In  Table  8  is  possible  to  compare  the  responses  to  P(A2)  according  each  interview 
taken  separately.  The  probabihty  assessment  has  its  lowest  value  of  0.68  from  interview 
number  3,  and  its  biggest  value  of  0.86  from  interview  number  4. 


Table  8.  Probabihty  Assessment  per  Interview 


MSTBQ  Responses  -  Class  A2 

Interview  n"  1 

2  3 

4 

Probability  Assessment 

P(A2)  -  Weighted  0.71 

0.82  0.68 

0.86 

The  average  taken  over  the  four  probabihty  assessment  is  0.77.  If  we  assume  that: 

•  The  average  value  taken  from  the  individual  results  of  each  interview  is 
appropriate  to  predict  P(A2),  and 

•  The  result  of  P(A2)  can  be  extrapolated  to  the  enhre  project,  that  is, 
implementation  of  SILOMS  in  the  MoD. 
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Then  the  average  taken  from  the  individual  scores  in  Table  8  can  be  interpreted  as  the 


probabihty  of  successful  implementation  of  SILOMS  in  the  MoD.  And  the  overall  result  is 
77%. 

On  the  other  hand,  the  results  from  interviews  three  and  four  show  considerable 
disagreements  in  the  responses  to  the  probability  assessment  of  class  A2.  For  instance,  if  we 
compare  the  results  obtained  from  these  interviews  (the  third  one  with  0.68,  and  the  fourth,  with 
0.86)  there  is  a  maximum  difference  of  0.18  in  the  probabihty  assessment. 

One  approach  to  solve  this  problem  is  to  use  the  so  called  “Delphi  Method”,  where  the 
results  from  individual  interviews  or  assessment  could  be  confronted  in  meetings  with  the 
participants  in  order  to  obtain  an  agreement  about  the  most  reasonable  response  to  the 
parameters  through  a  process  of  discussions  based  on  each  individual  experience  and  expertise. 
As  a  result  of  such  meetings,  the  agreed  scores  to  the  parameters  would  be  considered  the  most 
appropriate.  This  approach  was  not  use  in  this  research  due  to  time  constraints. 

It  is  also  interesting  to  note  that  the  scores  obtained  for  the  factors  in  interview  three 
shows  a  central  tendency,  that  is,  alternating  from  0.5  and  0.8  as  opposed  to  the  remaining  three 
interviews  that  were  scored  with  more  alternatives  within  the  Likerts  scale  (from  0.2  through 
1.0).  Then,  another  approach  to  deal  with  this  difference  is  to  not  consider  the  data  from 
respondent  three,  since  the  data  obtained  provides  httle  insight. 

If  only  interviews  one,  two,  and  four  are  considered,  the  average  obtained  in  those 
interviews,  will  be  0.79.  In  this  case,  the  probabihty  of  successful  implementation  of  SILOMS 
in  the  MoD  is  79%  compared  with  77%  taken  over  ah  interviews. 
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Sensitivity  Analysis  -  Force  Field  Analysis 


This  section  presents  the  force  field  analysis  for  the  responses  obtained  from  the 
MSTBQ.  The  results  were  used  as  a  basis  to  illustrate  the  data  shown  in  Table  7. 

Each  one  of  the  following  figures  represents  the  results  obtained  from  the  interviews 
taken  separately  and  identified  with  differentiated  dotted  arrows.  Also  a  resultant  force 
identified  by  a  non- dotted  arrow  was  calculated  using  the  simple  average  from  aU  four 
interview’s  results  for  the  parameters  being  considered. 
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Figure  12.  Force  Field  Analysis  to  Attributes  within  Organizational  Element  Risk 
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Figure  12  represents  the  force  field  analysis  taken  over  the  attributes  within  the 
Organizational  Risk  Element  (A2.4),  if  we  assume  that: 

•  The  simple  average  over  the  interviews’  response  is  appropriate  to  predict  the 
attributes’  probabihty  assessment,  and 

•  Each  attribute  has  the  same  weight  in  relation  with  the  result  over  P(A2.4). 
Then,  within  the  Organizational  Element  Risk  (A2.4),  the  attribute  Differences  in 

Organization’s  Structures  (A2.4.1)  is  the  one  that  requires  special  attention  from  the  project 
manager  since  it  has  the  higher  restraining  force  toward  project’s  failure. 

General  Force  Field  Analysis  of  Project  Performance 
A2.5  -  Cultural  Element  Risk 
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Figure  13.  Force  Field  Analysis  to  Attributes  within  Cultural  Element  Risk 


Figure  13  represents  the  force  field  analysis  taken  over  the  attributes  within  the  Cultural 
Element  Risk  (A2.5),  if  we  assume  that: 

•  The  simple  average  over  the  interviews’  response  is  appropriate  to  predict  the 
attributes’  probabihty  assessment,  and 
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Each  attribute  has  the  same  weight  in  relation  with  the  result  over  P(A2.5). 


Then,  within  the  Cultural  Element  Risk  (A2.5),  the  attribute  Willingness  to  Change 
(A2.5.2)  is  the  one  that  requires  special  attention  from  the  project  manager  since  it  has  the 
higher  restraining  force  toward  project’s  failure. 


General  Force  Field  Analysis  of  Project  Performance 
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Figure  14.  Force  Field  Analysis  to  Attributes  within  Pohtical  Element  Risk 


Figure  14  represents  the  force  field  analysis  taken  over  the  attributes  within  the  Political 
Element  Risk  (A2.6),  if  we  assume  that: 

•  The  simple  average  over  the  interviews’  response  is  appropriate  to  predict  the 
attributes’  probabihty  assessment,  and 

•  Each  attribute  has  the  same  weight  in  relation  with  the  result  over  P(A2.6). 
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Then,  within  the  Political  Element  Risk  (A2.5),  the  attribute  Feuds  Existence  in 


Organization’s  Politics  (A2.6.2)  is  the  one  that  requires  special  attention  from  the  project 
manager  since  it  has  the  higher  restraining  force  toward  project’s  failure. 


General  Force  Field  Analysis  of  Project  Performance 
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Figure  15.  Force  Field  Analysis  to  Risk  Elements  within  Program  Constraints  Class 


Figure  15  represents  the  force  field  analysis  taken  over  the  risk  elements  within  the 
Program  Constraints  Class  (A2),  if  we  assume  that: 

•  The  simple  average  over  the  interviews’  response  is  appropriate  to  predict 
each  elements’  probabihty  assessment,  and 

•  Each  element  has  the  same  weight  in  relation  with  the  result  over  P(A2). 
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Then,  within  the  Program  Constraints  Class  (A2),  the  parameter  Cultural  Element 


Risk  (A2.5)  is  the  one  that  requires  special  attention  from  the  project  manager  since  it  has  the 
higher  restraining  force  toward  project’s  failure.  Notice  ftiat  this  risk  element  has  the  higher 
restraining  force,  acting  toward  the  worst  state,  which  is  30%  against  the  project’s  success. 
This  can  be  seen  also  in  Table  7. 

The  same  force  field  analysis  could  have  been  used  after  considering  the  two 
approaches  suggested  in  the  previous  section,  that  is,  to  deal  with  the  data  gathered  from 
interview  number  three.  First,  in  that  case,  after  performing  the  “Delphi  Method”,  the  force  field 
analysis  would  only  consider  scores  obtained  in  the  agreement.  Second,  if  data  from  interview 
three  was  considered  not  reliable,  then  the  same  pictures  could  have  been  drawn  using  the  three 
selected  interviews. 

Investigative  Questions 

This  section  uses  the  analysis  performed  in  the  previous  sections  to  answer  each  one  of 
the  investigative  questions  “What  are  the  factors  critical  to  the  successful  implementation  of 
SILOMS  in  the  MoD?”  can  be  answered  by  the  use  of  MSTRI,  which  ehcit  the  factors  and 
considerations  that  may  turn  into  a  threat  to  the  implementation  of  SILOMS  in  the  MoD. 
Besides,  the  force  field  analysis  shows  how  sensitive  is  each  one  of  the  factors  or  parameter,  in 
relation  with  project’s  success. 

The  second  investigative  question  “What  is  an  appropriate  method  available  to  assess  or 
predict  risks  involved  in  the  implementation  of  SILOMS  in  the  MoD?”  can  be  answered  in  the 
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way  that  an  appropriate  method  is  the  use  of  MSTRI  and  its  derived  MSTQ  to  perform 


interviews  with  key  personnel  related  to  SILOMS  implementation  in  the  MoD. 

The  answer  to  the  third  investigative  question  “How  would  we  quantify  the  degree  of 
risks  in  order  to  help  the  decision  making  process  of  adopting  SILOMS  in  the  MoD?”  is  that 
the  use  of  a  combination  of  SERIM  and  Force  Field  Analysis  methodologies,  as  show  in  this 
research,  can  give  a  quantification  or  the  degree  of  risks  involved  in  the  implementation  of 
SIFOMS  in  the  MoD. 

The  answer  to  the  fourth  investigative  question  “Can  a  probabihty  of  success  be 
obtained  from  this  methodology?”  is  affirmative,  and  for  instance,  the  method  apphed  within  the 
scope  of  this  research,  showed  a  probabihty  of  approximately  77%  of  project’s  success, 
considering  the  Program  Constraints  Class  in  the  MSTRI. 

Chapter  Summary 

This  chapter  addressed  the  investigative  questions  described  in  Chapter  1  and  also 
presented  the  summary  of  scores  given  to  the  parameters  associated  with  the  Program 
Constraints  Class,  in  relation  to  the  implementation  of  SIFOMS  in  the  MoD.  Research  results 
obtained  through  the  use  of  MSTQ  and  foUowing  analyzes  and  interpretations  of  the  data  were 
presented.  FinaUy  the  results  from  the  force  field  analysis  were  presented. 
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V.  Conclusions 


Introduction 

This  chapter  synthesizes  the  findings  of  this  study.  In  the  first  section,  the  research 
question  wiU  be  revisited  and  conclusions  will  be  drawn  based  on  the  results  and  analysis 
performed  in  Chapter  IV.  The  seeond  section  describes  the  limitations  of  this  study.  The  third 
section  makes  recommendations  related  to  the  use  of  the  method  proposed  as  well  as  to  the 
successful  implementation  of  SILOMS  in  the  MoD.  The  fourth  seetion  wiU  point  out  issues  for 
future  research.  Finally  a  chapter  summary  will  be  presented. 

Conclusions 

After  analyzing  the  results  obtained  from  the  use  of  the  method  apphed  in  the 
implementation  of  SILOMS  in  the  MoD,  then,  in  this  point,  the  researeh  is  able  to  answer  the 
Research  Questions  stated  in  Chapter  1.  That  is  “How  to  assess  the  feasibility  and  risks  of  the 
implementation  of  SILOMS  in  the  MoD?” 

The  answer  comes  through  the  description  of  what  was  performed  so  far  in  the  research 
process: 

•  The  double  approach  in  the  research  design,  which  is  qualitative  and  quantitative 
studies,  has  been  used  as  a  way  to  compensate  the  gaps  that  exits  in  eaeh 
separately  approach. 

•  It  was  introduced  a  method  that  addresses  and  predict  the  risks  involved  in 
software  development  or  implementation  projects. 
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•  The  method  was  tested  in  the  case  of  the  implementation  of  SILOMS  in  the 
MoD,  limited  to  the  organizational,  cultural,  and  pohtical  aspects  that  can  threat 
the  project’s  success. 

•  The  proposed  method  provides  qualitative  and  quantitative  data  to  support  the 
MoD’s  decision  makers  in  evaluating  alternatives  available  for  the 
implementation  of  any  information  system  in  the  MoD. 

•  The  method  can  be  easily  extended  to  address  other  areas  of  risks  identified  in 
MSTRI,  and  then,  giving  a  better  judgment  about  the  risks  involved  in  the 
implementation  of  SILOMS  in  the  MoD. 

Also,  there  are  some  reasons  to  support  the  aforementioned  conclusions: 

•  The  method  was  tested  in  a  real-world  scenario,  and  despite  the  fact  that  was 
limited  for  a  few  aspects  of  the  MSRI  taxonomy,  the  results  proved  to  be  useful 
in  the  decision  making  process  or  deciding  over  the  best  alternative  available. 

•  The  method  has  provided  an  overall  assessment  of  the  probabihty  of  success 
involved  in  the  case  studied. 

•  The  method  is  fairly  easy  to  be  apphed. 

•  Given  the  importance  in  choosing  a  logistics  information  system  that  integrates 
the  supply  chain  management  in  the  MoD,  then  the  use  of  a  methodology  that 
deals  with  risks  and  probability  of  software  project’s  success  has  to  be  used  in 
the  evaluation  of  the  alternatives. 

Limitations 

The  method  was  only  apphed  considering  the  organizational,  cultural  and  pohtical 
aspects,  under  the  Program  Constraints  Class  -  MSTRI.  Also  the  weighting  process  was 
implemented  only  in  relation  of  the  elements  within  the  Program  Constraints  Class,  although  the 
method  could  have  been  used  to  consider  weights  in  any  level,  that  is,  every  factor,  attribute, 
element  and  class  considered  in  the  proposed  taxonomy. 
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Another  limitation  was  the  time  constraint  that  prevented  the  implementation  of  a 


procedure  to  minimize  disagreements  within  the  set  of  interviewers.  One  approach  could  be  to 
perform  a  “Delphi  Method”  in  order  to  minimize  those  disagreements  and  also  give  a  more 
rehable  overall  assessment  of  the  probabihty  of  success.  Another  approach  could  be  to  not 
consider  data  from  interviews  that  apparently  shows  some  sort  of  bias  or  not  plausible  explained 
tendency. 

Recommendations 

Since  the  method  was  tested  in  a  real-world  environment,  it  could  be  useful  to  extend 
the  method  to  cover  a  complete  assessment  of  SILOMS’  implementation  in  the  MoD. 

If  the  methodology  is  chosen  to  be  apphed,  then,  it  is  recommended  that  the  people  that 
will  conduct  the  interviews  and  tabulate  the  data  gathered  has  to  be  instmcted  in  detail  about 
how  the  method  works.  Also,  is  strongly  recommended  the  participation  of  SILOMS 
implementation’s  project  manager  in  the  process  of  choosing  the  main  parameters  and  in  the 
definition  of  the  sample  that  will  take  part  of  the  assessment. 

Also  is  strongly  recommended  that  futures  use  of  the  method  have  to  consider  other 
organizations  involved  in  SILOMS  due  to  the  fact,  that  such  an  integrated  system  has  the 
database  reliabihty  strongly  rehed  upon  lower  levels  of  management  and  operations.  These 
organizations  could  be  those  deahng  with  SILOMS  in  each  branch  of  military.  That  is,  the 
sample  used  to  perform  the  MSTBQ  have  to  consider  the  operational  or  end-users  in  the 
Brazihan’s  Army,  Navy  and  Air  Force,  in  order  to  get  an  overall  picture  of  the  risks  involved  in 
the  implementation  of  SILOMS  in  the  MoD. 
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Future  Research 


Future  research  could  be  the  test  of  the  proposed  methodology  to  aggregate  the  so- 
called  “Delphi  Method”  and  compare  the  differences  with  the  results  obtained  from  the  simple 
average  taken  over  the  scores  obtained  in  each  interview. 

Chapter  Summary 

This  chapter  has  synthesized  the  findings  of  this  study.  In  the  first  section,  the  research 
question  was  resembled  and  conclusions  were  drawn  based  on  the  results  and  analysis 
performed  in  Chapter  IV.  The  second  section  has  described  the  limitations  of  this  study  and  the 
third  section  presented  some  recommendations  related  to  the  use  of  the  method  proposed  as 
well  as  to  the  successful  implementation  of  SILOMS  in  the  MoD.  Finally  the  last  section 
pointed  out  issues  for  future  researches. 
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Appendix  A.  MoD-SILOMS  Taxonomy -Based  Questionnaire  -  MSTBQ 

Interview  Form 

This  questionnaire  was  developed  based  on  examples  and  methodologies  from  (1:A14- 
B24;  6:43-75)  and  according  researcher  experience  in  the  field. 

A.  Describing  SILOMS 

SILOMS  is  a  project  started  in  1993  aiming  to  achieve  an  integration  o  the  information 
systems  within  the  Brazilian  Air  Force  (FAB)  Materiel  Command  (COMGAP).  The  Integrated 
Systems  of  Logistics  Materiel  and  Services  (SILOMS)  integrates  in  a  single  corporate  database 
system  aU  logistics  information  related  to  maintenance,  supply,  and  transportation  within  the 
COMGAP.  The  overall  goal  of  the  system  is  to  provide  information  to  support  the  logistics 
decisions  makers  at  all  three  levels  within  COMGAP’ s  organizations  to  control  and  manage 
assets,  including  weapon  systems  and  related  equipment,  as  well  as  track  needs  during  systems’ 
life  cycle.  The  system  will  also  provide  a  clear  vision  of  the  movement  of  materials  within  the 
depots  and  related  bases.  Another  important  feature  of  the  system  is  to  allow  a  variety  of 
queries  in  the  corporate  database  to  collect  statistical  data  that  could  help  the  measurement  of 
key  performance  parameters  related  to  maintenance  activities  as  well  as  rehabihty  and 
availabihty  of  the  assets  being  controlled. 

B.  Purpose  of  the  Research 

ASSUMPTION:  There  is  a  need  for  a  logistics  information  system  in  the  MoD 

With  some  adaptations,  the  system  has  the  capabihty  to  fill  in  the  gap  that  exists  in  the 
MoD’s  Logistics  and  Mobilization  Agency  (SELOM),  by  allowing  integrated  management  of  aU 
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needs  within  the  mihtary  in  supporting  their  weapon  systems.  SILOMS  may  be  used,  for 
instance,  in  helping  identify  similar  parts  needed  by  all  defense  organizations  and  allowing 
SELOM  to  employ  a  consohdated  acquisition  of  supphes,  thereby  promoting  savings  and 
improving  the  efficiency  of  the  weapon  system  acquisition  process  and  their  associated  hfe 
cycle. 

The  objective  of  this  research  is  to  provide  a  method  to  measure  the  effort  and 
feasibihty  of  using  SILOM’s  functions  in  the  SELOM’ s  environment. 

a)  Critical  Issues  in  SILOMS  Implementation 

The  implementation  of  an  integrated  information  system  has  inherent  challenges. 
Differences  in  organization  culture,  or  in  the  way  tasks  are  performed,  are  key  issues  to  be 
observed  in  attempting  to  do  so.  The  same  is  applicable  when  trying  to  adapt  an  already 
existing  system  to  fill  in  the  need  of  another  organization.  In  such  new  environment,  a  key  issue 
is  to  assess  the  feasibihty  of  proceeding  with  an  adaptation  of  an  existing  information  system  or  if 
it  is  better  to  build  a  complete  new  system.  If  SELOM  chooses  to  use  the  SILOM,  what  has  to 
be  done  to  assure  the  success  of  its  implementation  in  MoD? 

C.  Demographic  Data  [Questions  I00-I05] 

[100]  What  is  your  rank  and  position  in  the  organization’ s  hierarchy? 

[101]  What  is  the  mission  of  the  organization  of  which  you  are  a  part? 

[102]  What  is  your  current  job? 

[103]  What  are  your  technical  quahfications? 

[103. a]  Do  you  have  a  background  in  logistics? 
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[103.b]  Do  you  have  a  background  in  System  Analysis  or  Software  Engineering? 


[104]  What  is  your  experience  (in  terms  of  years)  in  this  position  in  the  organization’s 
hierarchy? 

[105]  Have  you  worked  in  any  development  of  an  information  system? 

(Yes)  [105. a]  What  was  you  job? 

(No)  [105  .b]  Are  you  famihar  with  IS  development  process? 

First  Open-ended  Question  (Before  getting  the  “standard  data”  over  the  “elements” 
data)  [OEG  -  Open-Ended  Question] 

[OEG]  In  your  opinion,  based  on  your  background  and  this  scenario,  what  kind  of  problems 
or  issues  do  you  think  that  may  appear  in  such  attempt?  I  mean,  adapting  SIEOMS 
to  the  MoD  environment? 

D.  Scoring  Methods 

The  scoring  method  for  the  question  that  follows  this  section  was  based  on 
Karolak  in  Software  Engineering  Risk  Management.  Software  risk  metrics 
measure  items  associated  with  software  risk  factors  provide  an  indication  of 
software  risks  viewed  from  several  sources  of  information.  Using  metrics 
associated.  Software  risk  metrics  are  numeric  values  generated  from  questions. 

The  answers  to  the  questions  are  then  used  to  measure  the  characteristics  of  the 
software  risk  factors.  A  subjective  numeric  value  which  ranges  anywhere  from  0 
to  1  is  assigned  by  the  person  in  response  to  the  metric  question.  (6:51-52) 

Answers  to  the  questions  should  use  the  following  scale  as  a  general  reference: 


0 

0.2 

0.5 

0.8 

1.0 

None 

A  EMe 

Some 

Most 

All 

Eigure  16.  Scale  as  a  general  reference  -  Extracted  from  Karolak  (6:52) 
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E.  Program  Constraints  Class 

This  section  define  the  Program  Constraint  Class,  the  Elements  and  Attributes  as 
well  their  Factors  under  the  MSTRI,  which  identifies  the  risk  associated  with  software 
development  by  associating  questions  in  this  interview,  which  in  turn,  ^nerate  metrics  to 
measure  the  Factors,  Attributes,  Elements  to  get  an  overall  risk  assessment  of  the  Program 
Constraint  Class  related  to  the  implementation  of  SILOMS  in  the  MoD.  The  use  of  the  scale 
defined  above  helps  to  come  up  with  tables  that  relate  software  risk  metrics  to  the  intended 

Program  Constraint  Class  consists  of  the  “external”  of  the  project  -  the  factors  that 
are  outside  the  direct  control  of  the  project  but  can  still  have  major  effects  on  its  success. 
Program  constraints  include  the  following  elements  and  their  definitions: 

•  Organizational  elements  -  The  external  constraints  imposed  in  the  project  due  to 
differences  in  the  hierarchy/organochart  of  the  participating  organizations  interacting 
in  the  project. 

•  Cultural  elements  -  The  external  constraints  impose  in  the  project  due  to 
differences  between  the  participating  organizations,  in  the  “way  their  employees 
perceive  and  how  this  perception  creates  a  pattern  of  behefs,  values,  and 
expectations”  (Gibson:30). 

•  Political  elements  -  External  constraints  such  as  behavior  outside  the  legitimate, 
recognized  power  system,  designed  to  benefit  an  individual  or  subunit,  often  at  the 
expense  of  the  project  organization  in  general  or  designed  to  acquire  and  maintain 
the  power  or  “status  quo”  of  the  organizations  involved  in  the  project. 

•  Resources  elements  -  The  external  constraints  imposed  on  schedule,  staff,  budget, 
or  facUities. 

•  Contract  elements  -  The  terms  and  conditions  of  the  project  contract. 
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•  Program  interface  elements  -  The  external  interfaces  to  customers,  other 
contractors,  corporate  management,  and  vendors. 

Under  the  scope  of  this  research  and  due  to  the  fact  that  there  is  no  approved  project 

and/or  contract,  the  interview  wiU  only  be  related  to  the  first  three  elements,  that  is, 

organizational,  cultural  and  pohtical  elements  in  the  Program  Constraint  Class. 

Program  Constraints  Class  -  Questions 

The  following  three  sections  include  questions  that  are  used  to  measure  software 

development  risk  associated  with  “Program  Constraint  Class”  according  to  MSTRI. 

1.  Organizational  Elements  (Risk  Organizational)  -  A2.4 

The  following  questions  are  used  to  measure  the  software  development  risk  associated 

with  the  attributes  and  factors  related  to  “Organizational  Element  Risk”  under  the  “Program 

Constraint  Class”  according  to  MSTRI. 

Initial  Open-ended  Question  for  Organizational  Element  Data  [OEO] 

[OEO 1  ]  What  kind  of  problems  or  issues  could  you  foresee  if  you  were  supposed  to  use  a 
system  developed  by  the  Air  Eorce  and  consequently  refiecting  its  or^nizational 
structure? 

a.  Differences  in  Organizations  Stmctures  -  Attribute  (A2.4.1) 

A  value  of  0  indicates  that  the  organization’s  stmctures  differ  completely.  A  value  of 
0.5  indicates  there  are  some  differences  in  the  organizations  stmctures,  but  not  significantly.  A 
value  of  1  indicates  no  differences  in  the  organizations  stmctures. 
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rA2.4.l.l1  Do  you  think  that  other  branches  of  mihtary’s  organization  structures  differ 


significantly  from  your  branch? 


(  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

rA2.4.1.21  Do  you  think  that  this/these  differences  may  jeopardize  the  implementation  of 
SILOMS  in  the  MoD? 


{  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 

Obs:  The  scale  in  this  question  is  inverted,  that  is,  when  the  interviewed  answered  that 

he/she  strongly  agree  that  differences  in  organizations  stmctures  may  jeopardize  the 
implementation,  the  score  0  was  assigned,  and  when  he/she  strongly  agree  that  none 
of  differences  in  organizations  stmctures  may  jeopardize  the  implementation,  the 
score  1  where  assigned. 

rA2.4.l.31  Do  you  agree  that  despite  the  fact  that  may  exist  significantly  differences  in 

mihtary’s  organizational  stmctures,  the  implementation  of  SILOM  in  the  MoD 
can  be  successful? 


(  ) 

(  ) 

(  ) 

(  ) 

0 

0.2 

0.8 

1.0 

None 

A  Little 

Most 

All 
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2.4.1  -  Differences  in  Organizations  Structures  -  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1.1 

A2.4.1.2 

A2.4.1.3 

Attribute  Average  Final  Value 

Table  9.  Rank-Order  and  Weight  Process  to  Factors  in  the.  Attribute  “Differences  in 

Organizations  Structures”. 

b.  Managers  Commitment  to  Cross- Organizational  Projects-  Attribute! A2.4.2) 


A  value  of  0  indicates  that  Managers  Commitment  to  Cross- Organization’s  Projects  is 
not  perceived  by  the  interviewed.  A  value  of  0.5  indicates  that  in  some  cases,  Managers 
Commitment  to  Cross- Organization’s  Projects  is  easily  perceived.  A  value  of  1  indicates  full 
Managers  Commitment  to  Cross- Organization’s  Projects. 

[A2.4.2. 1]  When  you  were  working  with  other  mihtary  branch’s  personnel,  did  you  feel  that 
your  boss/senior  managers  were  committed  to  the  work/activity/project? 


{  ) 

(  ) 

(  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 

[A2.4.2.2]  Did/Do  you  feel  that  your  motivation  and  commitment  were/is  high  when  working 
with  other  mihtary  branch’s  personnel? 


{  ) 

(  ) 

(  ) 

{  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 
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[A2.4.2.3]  Would  you  describe  this  experience  as  a  enjoyable  experience? 


(  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

A2.4.2  Managers  Commitment  to  Cross  Organizational  Projects  -  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.2.1 

A2.4.2.2 

A2.4.2.3 

Attribute  Average  Final  Value 

Table  10.  Rank-Order  and  Weight  Process  to  Factors  in  the.  Attribute  “Managers 


Commitment  to  Cross- Organizational  Projects”. 

c.  Organization  Strategy  to  Cross- Organizational  Project  Management  - 
Attribute!  A2.4.3) 

A  value  of  0  indicates  there  is  no  documented  Organization  Strategy  to  Cross- 
Organizational  Project  Management.  A  value  of  0.5  indicates  that  there  is  no  documented 
Organization  Strategy  to  Cross- Organizational  Project  Management  but  managers  and 
employees  involved  in  such  activities  know  the  communication  lines  of  authority,  or  there  is  a 
documented  Organization  Strategy  to  Cross- Organizational  Project  Management  but  it  is  not 
correct/updated.  A  value  of  1  indicated  that  there  is  a  documented  Organization  Strategy  to 
Cross- Organizational  Project  Management  and  it  indicated  how  to  deal  with  this  kind  of 
activities. 
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rA2.4.3.l1  Does  your  organization  have  a  specific  written  strategy  to  deal  with  cross- 


organizational  projects?  (e.g.,  document,  statement  of  pohcy,  operating 
instmctions?) 


{  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

rA2.4.3.21  Do  you  think  that  this  kind  of  document/strategy  is  important  to  your  organization’s 
performance? 


(  ) 

(  ) 

{  ) 

{  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 

A2.4.3  Organization  Strategy  to  Cross-Organizational  Project  Management-  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.3.1 

A2.4.3.2 

Attribute  Average  Final  Value 

Table  1 1 .  Rank- Order  and  Weight  Process  to  Factors  in  the  Attribute  “Organization  Strategy 

to  Cross- Organizational  Project  Management”. 


Last  Open-ended  Question  to  Organizational  Elements  Data  [OEO] 

[0E02]  What  do  you  think  about  the  Brazihan  Air  Force  initiative  in  integrating  the  logistics 
functions  in  only  one  system? 
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Rank  Order  to  Organizational  Elements  Data  [ROO] 


[ROO]  If  you  were  asked  to  rank  order  the  previous  attributes,  from  the  most  important  to 
the  less  important,  how  it  should  be? 


A2.4  Organizational  Risk  -  Element 

Metric  -  Attribute 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1 

A2.4.2 

A2.4.3 

Attribute  Average  Final  Value 

Table  12.  Rank- Order  and  Weight  Process  to  Attributes  in  the  Element  “Organizational 


Risk”. 


2.  Cultural  Element  Risk  (Risk  Culture)  -  A2.5 

The  following  questions  are  used  to  measure  the  software  development  risk  associated 
with  the  attributes  and  factors  related  to  “Cultural  Element  Risk”  under  the  “Program  Constraint 
Class”  according  to  MSTRI. 

First  Open-ended  Question  to  Cultural  Elements  Data  [OEC] 

[OECl]  How  could  you  describe  the  culture  in  your  organization  and  your  department/agency? 
d.  Differences  in  Organizations  Cultures  -  Attribute(A2.5.1) 

A  value  of  0  indicates  that  the  organization’s  culture  differ  completely.  A  value  of  0.5 
indicates  there  are  some  differences  in  the  organizations  culture,  but  not  significantly.  A  value  of 
1  indicates  no  differences  in  the  organizations  cultures. 
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rA2.5.l.l1  Do  you  think  that  other  branches  of  mihtary’s  organization  cultures  differ 


significantly  from  your  branch? 


{  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

rA2.5.1.21  Do  you  think  that  this/these  differences  may  jeopardize  the  implementation  of 
SILOMS  in  the  MoD? 


(  ) 

(  ) 

(  ) 

{  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 

Obs:  The  scale  in  this  question  is  inverted,  that  is,  when  the  interviewed  answered  that  he/she 

strongly  agree  that  differences  in  organizations  cultures  may  jeopardize  the 
implementation,  the  score  0  was  assigned,  and  when  he/she  strongly  agree  that  none  of 
differences  in  organizations  cultures  may  jeopardize  the  implementation,  the  score  1 
where  assigned. 

rA2.5.l.31  Do  you  agree  that  despite  the  fact  that  may  exist  significantly  differences  in 

mihtary’s  organizational  cultures,  the  implementation  of  SILOM  in  the  MoD  can  be 
succesfuU? 


(  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 
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A2.5.1  Differences  in  Organizations  Cultures  -  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

2.5. 1.1 

2.5. 1.2 

2.5. 1.3 

Attribute  Average  Final  Value 

Table  13.  Rank- Order  and  Weight  Process  to  Factors  in  the  Attribute  “Differences  in 

Organizations  Cultures”. 


e.  Whlingness  to  Change  -  Attribute(A2.5.2) 

A  value  of  0  indicates  you  work  for  a  progressive  company,  which  is  constantly 
changing  in  its  decisions  and  culture.  A  value  of  0.5  indicates  you  work  for  a  moderately 
conservative  company,  which  needs  much  information  before  a  decision  is  made  or  tends  to 
perform/produce  activities/products  that  have  been  done  in  the  past.  A  value  of  1  indicates  you 
work  for  a  highly  innovative  company. 

rA2.5.2.11  Is  your  company/organization  culture  conservative  in  its  decision  making? 


{  ) 

(  ) 

{  ) 

{  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 

rA2.5.2.21  Does  your  company/organization  tend  to  build  or  acquire  new  products  and/or 


technologies? 


{  ) 


0 


( 


) 

0.2 


) 

0.5 


) 

0.8 


) 

1.0 


None 


A  Little 


Some 


Most 


All 
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A2.5.2  Willingness  to  Change  -  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

2.5.2.1 

2.5.2.2 

Attribute  Average  Final  Value 

Table  14.  Rank-Order  and  Weight  Process  to  Factors  in  the,  Attribute  “Willingness  to 

Change”. 


Last  Open-ended  Question  to  Cultural  Elements  Data  [OEC  -  Open-Ended  Question] 

[OEC2]  Do  you  think  that  exists  any  cultural  problems/issues  that  can  make  difficult  the 
implementation  of  SILOMS  in  the  MOD?  Do  you  think  that  exists  any  cultural 
aspect,  I  mean,  behefs,  patterns,  standards,  or  any  kind  of  behavior  within  your 
agency/department  that  may  turn  into  a  barrier  to  the  implementation  of  SILOMS  in 
the  MoD? 


Rank  Order  Question  to  Cultural  Elements  [ROC] 

[ROC]  If  you  were  asked  to  rank  order  these  (the  following)  issues,  from  the  most  important  to 
the  less  important,  how  it  should  be? 


A2.5  Cultural  Risk  -  Element 

Metric  -  Attribute 

Value 

Rank  Order 

Weight 

Final  Value 

A2.5.1 

A2.5.2 

Attribute  Average  Final  Value 

Table  15.  Rank- Order  and  Weigh  Process  to  Attributes  in  the  Element  “Cultural  Risk”. 
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3.  Political  Element  Risk  (Risk  Politics)  -  A2.6 


The  following  questions  are  used  to  measure  the  software  development  risk  associated 
with  the  attributes  and  factors  related  to  “Pohtical  Element  Risk”  under  the  “Program  Constraint 
Class”  according  to  MSTRI. 

First  Open-ended  Question  to  Political  Element  Data  [OEP] 

[OEPl]  Do  you  think  that  your  wiU  have  any  pohtical  problems/issues  in  your 

agency/department  if  the  ministry  gives  the  approval  to  implement  SIEOMS  in  the 
MoD? 
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f.  Internal  Disputes  in  Organizations  Politics  -  Attribute(A2.6.1) 

A  value  of  0  indicates  Internal  Disputes  in  Organizations’  Politics  occurs  frequently.  A 
value  of  0.5  indicates  that  Internal  Disputes  in  Organizations’  Politics  occurs  in  a  controllable 
way,  that  is,  not  affecting  the  organization’s  performance.  A  value  of  1  indicates  that  no  Internal 
Disputes  in  Organizations’  Politics  occurs. 


rA2.6.l.l1  What  kind  of  commitment  of  the  top-level  managers  are  you  expecting  if  they  were 
asked  to  implement  an  information  system  developed  by  the  Air  Force? 


rA2.6.1.21  If  you  were  asked  to  decide  about  the  implementation  of  SILOMS  in  your 
organization  would  you  approve  it? 


rA2.6.l.31  If  you  were  asked  to  decide  about  whether  choose  to  develop  your  own 

information  system  or  whether  to  adapt  and  already  existing  one,  would  you 
choose  SILOMS? 
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(  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

rA2.6.l.41  Would  you  agree  that  SILOMS,  a  system  used  by  Air  Force,  can  fulfill  the  needs 
of  you  agency/department  in  the  MoD? 


{  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 

rA2.6.l.51  If  you  were  asked  to  give  your  opinion  about  whether  to  use  a  COTS  or 
SILOMS,  would  you  recommend  SILOMS? 


{  ) 

(  ) 

{  ) 

{  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

A2.6.1  Internal  Disputes  in  Organizations  Politics  -  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.6.L1 

A2.6.L2 

A2.6.L3 

A2.6.L4 

A2.6.L5 

Attribute  Average  Final  Value 

Table  16.  Rank-Order  and  Weigh  Process  to  Factors  in  the  Attribute  “Internal  Disputes  in 

Organizations  Pohtics”. 
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g.  Feuds  Existence  in  Organizations  Politics  -  Attribute(2.6.2) 


A  value  of  0  indicates  that  Feuds  Existence  in  Organizations  Politics  highly  affects 
organization’s  performance.  A  value  of  0.5  indicates  that  Feuds  Existence  in  Organizations 
Pohtics  is  moderate  and  occurs  in  a  controllable  way,  that  is,  not  affecting  the  organization’s 
performance.  A  value  of  1  indicates  that  there  are  no  Eeuds  Existence  in  Organizations  Pohtics. 
rA2.6.2.l1  Does  good  communication  exist  between  different  organizations  supporting  the 
development  of  the  software  project? 


{  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Most 

All 

rA2.6.2.21  If  you  were  asked  to  give  your  opinion  about  the  different  smaU  groups  that  may 
exist  in  your  organization,  would  you  say  that  they  do  not  affects  the  organization’s 
performance? 


{  ) 

(  ) 

{  ) 

(  ) 

(  ) 

0 

0.2 

0.5 

0.8 

1.0 

None 

A  Little 

Some 

Most 

All 
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A2.6.2  Feuds  Existence  in  Organizations  Politics  -  Attribute 

Metric  -  Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.6.2.1 

A2.6.2.2 

Attribute  Average  Final  Value 

Table  17.  Rank-Order  and  Weigh  Process  to  Factors  in  the,  Attribute  “Feuds  Existence  in 

Organizations  Politics”. 


Last  Open-ended  Question  to  Political  Elements  Data  [OEP] 

[OEP2]  Do  you  think  that  would  exist  any  other  pohtical  problems/issues  in  the 
implementation  of  SIEOMS  in  the  MoD? 


Rank  Order  attributes  to  Political  Element  [ROP] 

[ROP]  If  you  were  asked  to  rank  order  these  (the  following)  issues,  from  the  most  important 
to  the  less  important,  how  it  should  be? 


A2.6  Political  Risk-  Element 

Metric  -  Attribute 

Value 

Rank  Order 

Weight 

Final  Value 

A2.6.1 

A2.6.2 

Attribute  Average  Final  Value 

Table  18.  Rank-Order  and  Weigh  Process  to  Attributes  the  Element  “Pohtical  Risk”. 
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Rank  Order  Elements  to  Program  Constraints  Class 


A2  Program  Constraints  -  Class 

Metric  -  Element 

Value 

Rank  Order 

Weight 

Final  Value 

A2.1 

A2.2 

A2.3 

A2.4 

A2.5 

A2.6 

Attribute  Average  Final  Value 

Table  19.  Rank-Order  and  Weigh  Process  to  Elements  in  the  Class  “Program  Constraints”. 
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Appendix  B.  SEI  Taxonomy  -  Program  Constraints  Class 


Program  constraints  refer  to  the  “externals”  of  the  project.  These  are  factors  that  may  be 
outside  the  control  of  the  project  but  can  still  have  major  effects  on  its  success  or  constitute 
sources  of  substantial  risk. 

1.  Resources 

This  Element  addresses  resources  for  which  the  program  is  dependent  on  factors  outside 
program  control  to  obtain  and  maintain.  These  include  schedule,  staff,  budget,  and  facihties. 

a)  Schedule 

This  attribute  refers  to  the  stabihty  of  the  schedule  with  respect  to  internal  and  external  events  or 
dependencies  and  the  viabihty  of  estimates  and  planning  for  all  phases  and  aspects  of  the 
program. 

b)  Staff 

This  attribute  refers  to  the  stabihty  and  adequacy  of  the  staff  in  terms  of  numbers  and  siU  levels, 
their  experience  and  skills  in  the  required  technical  areas  and  apphcation  domain,  and  teir 
availabihty  when  needed. 

c)  Budget 

This  attribute  refers  to  the  stabihty  of  the  budget  with  respect  to  internal  and  external  events  or 
dependencies  and  the  viabihty  of  estimates  and  planning  for  ah  phases  and  aspects  of  the 
program. 

d)  Facilities 

This  attribute  refers  to  the  adequacy  of  the  program  facihties  for  development,  integration,  and 
testing  of  the  product. 

2.  Contract 

Risks  associated  with  the  program  contract  are  classified  according  to  contract  type, 
restrictions,  and  dependencies. 

e)  Type  of  Contract 

This  attribute  covers  the  payment  terms  (cost  plus  aware  fee,  cost  plus  fixed  fee,  etc.)  and  the 
contractual  requirements  associated  with  such  items  as  the  Statement  of  Work,  Contract  Data, 
Requirements  List,  and  the  amount  and  conditions  of  customer  involvement. 
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f)  Restrictions 

Contract  restrictions  and  restrints  refer  to  contractual  directives  to,  for  example,  use  specific 
development  methods  or  equipment  and  the  resultant  complications  such  as  acquisition  of  data 
rights  for  use  of  non- developmental  software. 

g)  Dependencies 

This  attribute  refers  to  the  possible  contractual  dependencies  on  outside  contractors  or  vendors, 
customers- furnished  equipment  or  software,  or  other  outside  products  and  services. 

3.  Program  Interfaces 

This  element  consists  of  the  various  interfaces  with  entities  and  organizations  outside  the 
development  program  itself. 

h)  Customer 

The  customer  attribute  refers  to  the  customer’s  level  of  skill  and  experience  in  the  technical  or 
application  domain  of  the  program  as  well  as  difficult  working  relationships  or  poor  mechanisms 
for  attaining  customer  agreement  and  approvals,  not  having  access  to  certain  customer  factions, 
or  not  being  able  to  communicate  with  the  customer  in  a  forthright  manner. 

i)  Associate  Contractors 

The  presence  of  associate  contractors  may  introduce  risks  due  to  conflicting  political  agendas, 
prolems  of  interfaces  to  systems  being  developed  by  outside  organizations,  or  lack  of 
cooperation  in  coordinating  schedules  and  configuration  changes. 

j)  Subcontractors 

The  presence  of  subcontractors  may  introduce  risks  due  to  inadequate  task  definitions  and 
subcontractor  management  mechanisms,  or  to  not  transferring  subcontractor  technology  and 
knowledge  to  the  program  or  corporation. 

k)  Prime  Contractor 

When  the  program  is  a  subcontract,  risks  may  arise  from  poorly  defined  task  definitions, 
complex  reporting  arrangements,  or  dependencies  on  technical  or  programmatic  information. 

l)  Corporate  Management 

Risks  in  the  corporate  management  area  include  poor  communication  and  direction  from  senior 
management  as  well  as  non- optimum  levels  of  support. 

m)  Vendors 

Vendor  risks  may  present  themselves  in  the  forms  of  dependencies  on  deliveries  and  support  for 
critical  system  components. 
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n)  Politics 

Political  risks  may  accme  from  relationships  with  the  company,  customer,  associate  contractors 
or  subcontractors,  and  may  affect  technical  decisions. 


92 


Appendix  C  SEI  TBQ  -  Program  Constraints  Class 

1.  Resources 

a.  Schedule 

[Is  the  schedule  inadequate  or  unstable?] 

[143]  Has  the  schedule  been  stable? 

[144]  Is  the  schedule  realistic? 

(Yes)  (144.a)  Is  the  estimation  method  based  on  historical  data? 

(Yes)  (144.b)  Has  the  method  worked  well  in  the  past? 

[145]  Is  there  anything  for  which  adequate  schedule  was  not  planned? 

•  Analysis  and  studies 

•  QA 

•  Training 

•  Maintenance  courses  and  training 

•  Capital  equipment 

•  Deliverable  development  system 

[146]  Are  there  external  dependencies  which  are  hkely  to  impact  the  schedule? 

b.  Staff 

[Is  the  staff  inexperienced,  lacking  domain  knowledge,  lacking  skills,  or 
understaffed?] 

[147]  Are  there  any  areas  in  which  the  required  technical  skills,  or  understaffed? 

•  Software  engineering  and  requirements  analysis  method 

•  Algorithm  expertise 

•  Design  and  design  methods 

•  Programming  languages 

•  Integration  and  test  methods 

•  Rehability 

•  Maintainability 

•  Availability 

•  Human  factors 

•  Configuration  mana^ment 

•  Quahty  assurance 

•  Target  environment 

•  Level  of  security 

•  COTS 

•  Reuse  software 
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•  Operating  system 

•  Database 

•  Application  domain 

•  Performance  analysis 

•  Tune- critical  applications 

[148]  Do  you  have  adequate  personnel  to  staff  the  program? 

[149]  Is  the  staffing  stable? 

[150]  Do  you  have  access  to  the  right  people  when  you  need  them? 

[151]  Have  the  program  members  implemented  systems  of  this  type? 

[152]  Is  the  program  rehant  on  a  few  key  people? 

[153]  Is  there  any  problem  with  getting  cleared  people? 

c.  Budget 

[Is  the  funding  insufficient  or  unstable?] 

[154]  Is  the  budget  stable? 

[155]  Is  the  budget  based  on  a  realistic  estimate? 

(Yes)  (155.a)  Is  the  estimation  method  based  on  historical  data? 

(Yes)  (155.b)  Has  the  method  worked  well  in  the  past? 

[156]  Have  features  or  functions  been  deleted  as  a  part  of  a  design- to- cost  effort? 

[157]  Is  there  anything  for  which  adequate  budget  was  not  allocated? 

•  Analysis  and  studies 

•  QA 

•  Training 

•  Maintenance  courses 

•  Capital  equipment 

•  Dehverable  development  system 

[158]  Do  budget  changes  accompany  requirement  changes? 

(Yes)  (158. a)  Is  this  a  standard  part  of  the  change  control  process? 

d.  Facihties 

[Are  the  facilities  adequate  for  building  and  delivering  the  product?] 

[159]  Are  the  development  facilities  adequate? 

[160]  Is  the  integration  environment  adequate? 

2.  Contract 

e.  Type  of  Contract 

[Is  the  contract  type  a  source  of  risk  to  the  program?  ] 
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[161]  What  type  of  contract  do  you  have?  (Cost  plus  award  fee,  fixed  price,. . .) 

(Yes)  (16  La)  Does  this  present  any  problems? 

[162]  Is  the  contract  burdensome  in  any  aspect  of  the  program? 

•  SOW  (Statement  of  Work) 

•  Specifications 

•  DIDs  (Data  Item  Descriptions) 

•  Contract  Parts 

•  Excessive  customer  involvement 

[163]  Is  required  documentation  burdensome? 

•  Excessive  amount 

•  Picky  customer 

•  Long  approval  cycle 

f  Restrictions 

[Does  the  contract  cause  any  restrictions?] 

[164]  Are  the  problems  with  data  rights? 

•  COTS  software 

•  Developmental  software 

•  Non- developmental  items 

g.  Dependencies 

[Does  the  program  have  any  dependencies  on  outside  products  or  services?] 

[165]  Are  there  dependencies  on  external  products  or  services  that  may  affect  the  product, 
budget,  or  schedule? 

•  Associate  contractors 

•  Prime  contractor 

•  Subcontractors 

•  Vendors  or  suppliers 

•  Customer  furnished  equipment  or  software 

3.  Program  Interfaces 

h.  Customer 

[Are  there  any  customer  problems  such  as:  lengthy  document -approval  cycle, 
poor  communication,  and  inadequate  domain  expertise  ? [ 

[166]  Is  the  customer  approval  cycle  timely  ? 

•  Documentation 

•  Program  reviews 

•  Eormal  reviews 

[167]  Do  you  ever  proceed  before  receiving  customer  approval? 
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[168]  Does  the  customer  understand  the  technical  aspects  of  the  system? 

[169]  Does  the  customer  understand  software? 

[170]  Does  the  customer  interfere  with  process  or  people? 

[171]  Does  management  work  with  the  customer  to  reach  mutually  agreeable  decisions  in  a 
timely  manner? 

•  Requirements  understanding 

•  Test  criteria 

•  Schedule  adjustments 

•  Interfaces 

[172]  How  effective  are  your  mechanisms  for  reaching  agreements  with  the  customers? 

•  Working  groups  (contractual?) 

•  Technical  interchange  meetings  (contractual?) 

[173]  Are  all  customers  factions  involved  in  reaching  agreements? 

(Yes)  (173. a)  Is  is  a  formally  defined  process? 

[174]  Does  management  present  a  realistic  or  optimistic  picture  to  the  customer? 

K  there  are  associate  contractors 

i.  Associate  Contractors 

[Are  there  any  problems  with  associate  contractors  such  as  inadequately  defined 
or  unstable  interfaces,  poor  communications,  or  lack  of  cooperation?] 

[175]  Are  there  external  interfaces  changing  without  adequate  notification,  coordination,  or 
formal  change  procedures? 

[176]  Is  there  and  adequate  transition  plan? 

(Yes)  (176.a)  Is  it  supported  by  all  contractors  and  site  personnel? 

[177]  Is  there  any  problem  with  getting  schedules  or  interface  data  from  associate  contractors? 
(No)  (177. a)  Are  they  accurate? 

K  there  are  subcontractors 

j.  Subcontractors 

[Is  the  program  dependent  on  subcontractors  for  any  critical  areas?] 

[178]  Are  there  any  ambiguities  in  subcontractors  task  definitions? 

[179]  Is  the  subcontractor  reporting  and  monitoring  procedure  different  from  the  program’s 
reporting  requirements? 

[180]  Is  subcontractor  administration  and  technical  management  done  by  a  separate 
organization? 

[181]  Are  you  highly  dependent  on  subcontractor  expertise  in  any  areas? 

[182]  Is  subcontractor  knowledge  being  transferred  to  the  company? 

[183]  Is  there  any  problem  with  getting  schedules  or  interface  data  from  subcontractors? 
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ff  program  is  a  subcontract 


k.  Prime  Contractor 

[Is  the  program  facing  difficulties  with  its  Prime  contractor?] 

[184]  Are  your  task  definitions  from  the  Prime  contractor  ambiguous? 

[185]  Do  you  interface  with  two  separate  prime  organizations  for  administrations  and  technical 
management? 

[186]  Are  you  highly  dependent  on  the  Prime  for  expertise  in  any  areas? 

[187]  Is  there  any  problem  with  getting  schedules  or  interface  data  from  the  Prime? 

l.  Corporate  Management 

[Is  there  a  lack  of  support  of  micro  management  form  upper  management?] 

[188]  Does  program  management  communicate  problems  to  senior  management? 

(Yes)  (188. a)  Does  this  seem  to  be  effective? 

[189]  Does  corporate  management  give  you  timely  support  in  solving  your  problems? 

[190]  Does  corporate  management  tend  to  micro- manage? 

[191]  Does  management  present  a  realistic  or  optimistic  picture  to  senior  management? 

m  Vendors 

]Are  vendors  responsive  to  program  needs?] 

[192]  Are  you  relying  on  vendors  for  deliveries  of  critical  components? 

•  Compilers 

•  Hardware 

•  COTS 

a  Politics 

]Are  politics  causing  a  problem  for  the  program?] 

[193]  Are  politics  affecting  the  program? 

•  Company 

•  Customer 

•  Associate  contractors 

•  Subcontractors 

[194]  Are  politics  affecting  technical  decisions? 
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Appendix  D.  Scores  Obtained  in  interviews  -  MSTBQ  -  Weighted  Scores 


Summary  of  Scores  from  Interview  #  1 
Organizational  Element  Risk 

2.4.1  -  Differences  in  Organizations  Structures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1.1 

0.5 

1 

1 

0.5 

A2.4.1.2 

0.2 

1 

1 

0.2 

A2.4.1.3 

0.8 

1 

1 

0.8 

Attribute  Average  Final  Value 

0.5 

A2.4.2  Managers  Commitment  to  Cross  Organizational  Projects  -  Attribute 


Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.2.1 

0.8 

1 

1 

0.8 

A2.4.2.2 

0.8 

1 

1 

0.8 

A2.4.2.3 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.866666667 

A2.4.3  Organization  Strategy  to  Cross-Organizational  Project  Management- 

Attribute 


Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.3.1 

0.5 

1 

1 

0.5 

A2.4.3.2 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.75 

A2.4  Organizational  Risk  -  Element 


Metric  - 

Attribute 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1 

0.5 

3 

0.6 

0.3 

A2.4.2 

0.866666667 

2 

0.9 

0.78 

A2.4.3 

0.75 

1 

1.5 

1.125 

1 

Element  Average  Final  Value  #  1 

0.735 

Summary  of  Scores  from  Interview  #  1 


Cultural  Element  Risk 


A2.5.1  Differences  in  Organizations  Cultures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5. 1.1 

0.5 

1 

1 

0.5 

2.5. 1.2 

0.2 

1 

1 

0.2 

2.5. 1.3 

1 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.566666667 

A2.5.2  Willingness  to  Change  -  Attribute 

Metric  - 
Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5.2.1 

0.2 

1 

1 

0.2 

2.5.22 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Final  Value 

0.5 

A2.5  Cultural  Risk  -  Element 

Metric  - 
Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.5.1 

0.566666667 

2 

0.8 

0.453333333 

A2.5.2 

0.5 

1 

1.2 

0.6 

1 

Element  Average  Einal  Value  #  1 

0.526666667 
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Summary  of  Scores  from  Interview  #  1 
Political  Element  Risk 


A2.6.1  Internal  Disputes  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1.1 

0.8 

1 

1 

0.8 

A2.6.1.2 

1 

1 

1 

1 

A2.6.1.3 

1 

1 

1 

1 

A2.6.1.4 

0.8 

1 

1 

0.8 

A2.6.1.5 

1 

1 

1 

1 

1 

Attribute  Average  Einal  Value 

0.92 

A2.6.2  Feuds  Existence  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.2.1 

0.5 

1 

1 

0.5 

A2.6.2.2 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Einal  Value 

0.65 

A2.6  Political  Risk-  Element 

Metric  - 
Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1 

0.92 

2 

0.8 

0.736 

A2.6.2 

0.65 

1 

1.2 

0.78 

1 

Element  Average  Einal  Value  #  1 

0.758 
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Summary  of  Scores  from  Interview  #  2 
Organizational  Element  Risk 

2.4.1  -  Differences  in  Organizations  Structures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1.1 

0.2 

1 

1 

0.2 

A2.4.1.2 

0.5 

1 

1 

0.5 

A2.4.1.3 

1 

1 

1 

1 

Attribute  Average  Final  Value  0.566666667 


A2.4.2  Managers  Commitment  to  Cross  Organizational  Projects  -  Attribute 

Metric  -  Value  Rank  Order  Weight  Final  Value 

Factor _ 

A2.4.2.1 _ 1 _ 1 _ 1 _ 1 

A2.4.2.2 _ 1 _ 1 _ 1 _ 1 

A2.4.2.3  11  11 


_ I _ Attribute  Average  Final  Value] _ 1 

A2.4.3  Organization  Strategy  to  Cross-Organizational  Project  Management- 

Attribute 


Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.3.1 

0.8 

1 

1 

0.8 

A2.4.3.2 

1 

1 

1 

1 

: 

Attribute  Average  Final  Value 

0.9 

A2.4  Organizational  Risk  -  Element 


Metric  - 

Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.4.1 

0.566666667 

2 

0.9 

0.51 

A2.4.2 

1 

1 

1.5 

1.5 

A2.4.3 

0.9 

3 

0.6 

0.54 

1 

Element  Average  Einal  Value  #  2 

0.85 
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Summary  of  Scores  from  Interview  #  2 


Cultural  Element  Risk 


A2.5.1  Differences  in  Organizations  Cultures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5. 1.1 

0.2 

1 

1 

0.2 

2.5. 1.2 

0.8 

1 

1 

0.8 

2.5. 1.3 

1 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.666666667 

A2.5.2  Willingness  to  Change  -  Attribute 

Metric  - 
Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5.2.1 

0.5 

1 

1 

0.5 

2.5.22 

1 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.75 

A2.5  Cultural  Risk  -  Element 

Metric  - 
Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.5.1 

0.666666667 

2 

0.8 

0.533333333 

A2.5.2 

0.75 

1 

1.2 

0.9 

1 

Element  Average  Einal  Value  #  2 

0.716666667 
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Summary  of  Scores  from  Interview  #  2 
Political  Element  Risk 


A2.6.1  Internal  Disputes  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1.1 

1 

1 

1 

1 

A2.6.1.2 

1 

1 

1 

1 

A2.6.1.3 

0.8 

1 

1 

0.8 

A2.6.1.4 

0.8 

1 

1 

0.8 

A2.6.1.5 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Einal  Value 

0.88 

A2.6.2  Feuds  Existence  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.2.1 

0.8 

1 

1 

0.8 

A2.6.2.2 

1 

1 

1 

1 

1 

Attribute  Average  Einal  Value 

0.9 

A2.6  Political  Risk-  Element 

Metric  - 
Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1 

0.88 

1 

1.2 

1.056 

A2.6.2 

0.9 

2 

0.8 

0.72 

1 

Element  Average  Einal  Value  #  2 

0.888 
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Summary  of  Scores  from  Interview  #  3 
Organizational  Element  Risk 

2.4.1  -  Differences  in  Organizations  Structures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1.1 

0.5 

1 

1 

0.5 

A2.4.1.2 

0.5 

1 

1 

0.5 

A2.4.1.3 

0.8 

1 

1 

0.8 

Attribute  Average  Final  Value 

0.6 

A2.4.2  Managers  Commitment  to  Cross  Organizational  Projects  -  Attribute 


Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.2.1 

0.8 

1 

1 

0.8 

A2.4.2.2 

0.5 

1 

1 

0.5 

A2.4.2.3 

0.8 

1 

1 

0.8 

Attribute  Average  Final  Value 

0.7 

A2.4.3  Organization  Strategy  to  Cross-Organizational  Project  Management- 

Attribute 


Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.3.1 

0.5 

1 

1 

0.5 

A2.4.3.2 

0.8 

1 

1 

0.8 

Attribute  Average  Final  Value 

0.65 

A2.4  Organizational  Risk  -  Element 


Metric  - 

Attribute 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1 

0.6 

1 

1.5 

0.9 

A2.4.2 

0.7 

2 

0.9 

0.63 

A2.4.3 

0.65 

3 

0.6 

0.39 

1 

Element  Average  Final  Value  #  3 

0.64 
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Summary  of  Scores  from  Interview  #  3 
Cultural  Element  Risk 


A2.5.1  Differences  in  Organizations  Cultures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5. 1.1 

0.8 

1 

1 

0.8 

2.5. 1.2 

0.5 

1 

1 

0.5 

2.5. 1.3 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Final  Value 

0.7 

A2.5.2  Willingness  to  Change  -  Attribute 

Metric  - 
Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5.2.1 

0.5 

1 

1 

0.5 

2.5.22 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Final  Value 

0.65 

A2.5  Cultural  Risk  -  Element 

Metric  - 
Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.5.1 

0.7 

2 

0.8 

0.56 

A2.5.2 

0.65 

1 

1.2 

0.78 

1 

Element  Average  Einal  Value  #  3 

0.67 
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Summary  of  Scores  from  Interview  #  3 
Political  Element  Risk 


A2.6.1  Internal  Disputes  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1.1 

0.8 

1 

1 

0.8 

A2.6.1.2 

0.8 

1 

1 

0.8 

A2.6.1.3 

0.8 

1 

1 

0.8 

A2.6.1.4 

0.5 

1 

1 

0.5 

A2.6.1.5 

0.5 

1 

1 

0.5 

1 

Attribute  Average  Einal  Value 

0.68 

A2.6.2  Feuds  Existence  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.2.1 

0.8 

1 

1 

0.8 

A2.6.2.2 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Einal  Value 

0.8 

A2.6  Political  Risk-  Element 

Metric  - 
Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1 

0.68 

1 

1.2 

0.816 

A2.6.2 

0.8 

2 

0.8 

0.64 

1 

Element  Average  Einal  Value  #  3 

0.728 
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Summary  of  Scores  from  Interview  #  4 
Organizational  Element  Risk 

2.4.1  -  Differences  in  Organizations  Structures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.1.1 

0.8 

1 

1 

0.8 

A2.4.1.2 

0.2 

1 

1 

0.2 

A2.4.1.3 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.666666667 

A2.4.2  Managers  Commitment  to  Cross  Organizational  Projects  -  Attribute 

Metric  -  Value  Rank  Order  Weight  Final  Value 

Factor 

A2.4.2.1 _ 1 _ 1 _ 1 _ 1 

A2.4.2.2 _ 1 _ 1 _ 1 _ 1 

A2.4.2.3  I  1  ~  1  I  1  _ 1 

_ Attribute  Average  Final  Value _ 1 

A2.4.3  Organization  Strategy  to  Cross-Organizational  Project  Management- 

Attribute 


Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Final  Value 

A2.4.3.1 

0.5 

1 

1 

0.5 

A2.4.3.2 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.75 

A2.4  Organizational  Risk  -  Element 


Metric  - 

Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.4.1 

0.666666667 

2 

0.9 

0.6 

A2.4.2 

1 

1 

1.5 

1.5 

A2.4.3 

0.75 

3 

0.6 

0.45 

1 

Element  Average  Einal  Value  #  4 

0.85 
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Summary  of  Scores  from  Interview  #  4 


Cultural  Element  Risk 


A2.5.1  Differences  in  Organizations  Cultures  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5. 1.1 

0.8 

1 

1 

0.8 

2.5. 1.2 

0.8 

1 

1 

0.8 

2.5. 1.3 

1 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.866666667 

A2.5.2  Willingness  to  Change  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

2.5.2.1 

0.8 

1 

1 

0.8 

2.5.2.2 

1 

1 

1 

1 

1 

Attribute  Average  Final  Value 

0.9 

A2.5  Cultural  Risk  -  Element 

Metric  - 

Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.5.1 

0.866666667 

2 

0.8 

0.693333333 

A2.5.2 

0.9 

1 

1.2 

1.08 

1 

Element  Average  Einal  Value  #  4 

0.886666667 
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Summary  of  Scores  from  Interview  #  4 
Political  Element  Risk 


A2.6.1  Internal  Disputes  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1.1 

1 

1 

1 

1 

A2.6.1.2 

1 

1 

1 

1 

A2.6.1.3 

1 

1 

1 

1 

A2.6.1.4 

0.8 

1 

1 

0.8 

A2.6.1.5 

1 

1 

1 

1 

1 

Attribute  Average  Einal  Value 

0.96 

A2.6.2  Feuds  Existence  in  Organizations  Politics  -  Attribute 

Metric  - 

Factor 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.2.1 

0.8 

1 

1 

0.8 

A2.6.2.2 

0.8 

1 

1 

0.8 

1 

Attribute  Average  Einal  Value 

0.8 

A2.6  Political  Risk-  Element 

Metric  - 

Attribute 

Value 

Rank  Order 

Weight 

Einal  Value 

A2.6.1 

0.96 

1 

1.2 

1.152 

A2.6.2 

0.8 

2 

0.8 

0.64 

1 

Element  Average  Einal  Value  #  4 

0.896 
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Appendix  E.  SERIM  Method  -  Calculations  applied  to  the  implementation  of 


SILOMS  in  the  MoD 


Class 


Elements 


Organizational 


3 


Attribute 


Attribute 


Attribute 


- P(A24)  :=  V  Wn(A2n) 

n  =  1 

3 

- P(A241)  :=  V  Wr<A241n) 

n=  1 

3 

- P(A242)  :=  V  Wn(A241n) 

n  =  1 

2 

- P(A243)  :=  V  Wn(A241n) 

n=  1 


Program  Constraints 

6 

P(A2)  :=  V  Wr<A2n) 
n  =  4 

Cuitural 


2 

- P(A25)  :=  V  Wt<A25n) 

n=  1 

3 

- P(A251)  :=  V  Wr<A251n) 

n=  1 

2 

- P(A252)  :=  V  Wn(;A252n) 

n=  1 


Political 

2 

- P(A26)  :=  V  Wr<A26n) 

n  =  1 

5 

- P(A261)  :=  V  Wr<A261n) 

n=  1 

2 

- PCA262)  :=  V  Wn(A262n) 

n=  1 


Figure  17.  Formulas  based  on  SERIM  Method  (6:121-131) 
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